{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "American_Option",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyN1hJHbW6gEOIkXtIJA/R9P",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/letianzj/QuantResearch/blob/master/ml/american_option.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LPsT-tou7RVo",
        "colab_type": "text"
      },
      "source": [
        "### Introduction\n",
        "\n",
        "This notebook uses reinforcement learning to price American option assuming flat volatility surface. \n",
        "\n",
        "It has three sections, corresponding to three packages used.\n",
        "\n",
        "1. Analytical baseline and QuantLib: to price European option and American option as reference values.\n",
        "\n",
        "2. Gym and Monte Carlo simulation: to create OpenAI gym environment.\n",
        "\n",
        "3. TF-Agent and Optimal stopping policy: to training a reinforcement learning agent and policy."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6YIxnvS-9oJE",
        "colab_type": "text"
      },
      "source": [
        "### Section One -- Baseline"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lmXsqxw9_42S",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "9a184c0e-db5a-4579-fa5d-d69544525811"
      },
      "source": [
        "# !pip install QuantLib-Python"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting QuantLib-Python\n",
            "  Downloading https://files.pythonhosted.org/packages/7e/a9/4c6fac9c3c9b1625ab573531bbe9666d2f7cd6bc0a5eeb1f5792b947d9b2/QuantLib_Python-1.18-py2.py3-none-any.whl\n",
            "Collecting QuantLib\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/02/48/cd7c50ee6fd0681fb9c007052ff50909050fac01ffd1d211e52df43c3af6/QuantLib-1.19-cp36-cp36m-manylinux1_x86_64.whl (18.5MB)\n",
            "\u001b[K     |████████████████████████████████| 18.5MB 1.3MB/s \n",
            "\u001b[?25hInstalling collected packages: QuantLib, QuantLib-Python\n",
            "Successfully installed QuantLib-1.19 QuantLib-Python-1.18\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D-pGm_YPAs-b",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import QuantLib as ql \n",
        "\n",
        "maturity = ql.Date(31, 12, 2019)\n",
        "S0 = 100\n",
        "K = 100\n",
        "r = 0.02\n",
        "sigma = 0.20\n",
        "d =  0.0\n",
        "otype = ql.Option.Put\n",
        "dc = ql.Actual365Fixed()\n",
        "calendar = ql.NullCalendar()\n",
        "\n",
        "today = ql.Date(1, 1, 2019)\n",
        "ql.Settings.instance().evaluationDate = today"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v1UQkRyuA2qJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "payoff = ql.PlainVanillaPayoff(otype, K)\n",
        "\n",
        "european_exercise = ql.EuropeanExercise(maturity)\n",
        "european_option = ql.VanillaOption(payoff, european_exercise)\n",
        "\n",
        "american_exercise = ql.AmericanExercise(today, maturity)\n",
        "american_option = ql.VanillaOption(payoff, american_exercise)"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sn83i6ZrA5Ld",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "d_ts = ql.YieldTermStructureHandle(ql.FlatForward(today, d, dc))\n",
        "r_ts = ql.YieldTermStructureHandle(ql.FlatForward(today, r, dc))\n",
        "sigma_ts = ql.BlackVolTermStructureHandle(ql.BlackConstantVol(today, calendar, sigma, dc))\n",
        "bsm_process = ql.BlackScholesMertonProcess(ql.QuoteHandle(ql.SimpleQuote(S0)), d_ts, r_ts, sigma_ts)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LWW5XQueA7jG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "f9d14c17-cdad-44cd-f773-f924fc737a51"
      },
      "source": [
        "pricing_dict = {}\n",
        "\n",
        "bsm73 = ql.AnalyticEuropeanEngine(bsm_process)\n",
        "european_option.setPricingEngine(bsm73)\n",
        "pricing_dict['BlackScholesEuropean'] = european_option.NPV()\n",
        "\n",
        "analytical_engine = ql.BaroneAdesiWhaleyEngine(bsm_process)\n",
        "american_option.setPricingEngine(analytical_engine)\n",
        "pricing_dict['BawApproximation'] = american_option.NPV()\n",
        "\n",
        "binomial_engine = ql.BinomialVanillaEngine(bsm_process, \"crr\", 100)\n",
        "american_option.setPricingEngine(binomial_engine)\n",
        "pricing_dict['BinomialTree'] = american_option.NPV()\n",
        "\n",
        "# fd_engine = ql.FdBlackScholesVanillaEngine(bsm_process)\n",
        "# american_option.setPricingEngine(fd_engine)\n",
        "# pricing_dict['FiniteDifference'] = american_option.NPV()\n",
        "\n",
        "print(pricing_dict)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{'BlackScholesEuropean': 6.92786901829998, 'BawApproximation': 7.091254636695334, 'BinomialTree': 7.090924645858217}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1nTMuUEkp2wN",
        "colab_type": "text"
      },
      "source": [
        "European option price is \\$6.928; American Option is \\$7.091; The value of early exercise is \\$0.163."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TklAYRUS-i8b",
        "colab_type": "text"
      },
      "source": [
        "### Section Two -- Gym Environment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vy2qUsT7PM1t",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import gym\n",
        "class AmeriOptionEnv(gym.Env):\n",
        "    def __init__(self):\n",
        "        self.S0 = 100.0\n",
        "        self.K = 100.0\n",
        "        self.r = 0.02\n",
        "        self.sigma = 0.20\n",
        "        self.T = 1.0\n",
        "        self.N = 365    # 365 day\n",
        "\n",
        "        self.S1 = 0\n",
        "        self.reward = 0\n",
        "        self.day_step = 0    # from day 0 taking N steps to day N\n",
        "\n",
        "        self.action_space = gym.spaces.Discrete(2)         # 0: hold, 1:exercise\n",
        "        self.observation_space = gym.spaces.Box(low=np.array([0, 0]), high=np.array([np.inf, 1.0]), dtype=np.float32)      # S in [0, inf], tao in [0, 1]\n",
        "\n",
        "    def step(self, action):\n",
        "        if action == 1:        # exercise\n",
        "            reward = max(K-self.S1, 0.0) * np.exp(-self.r * self.T * (self.day_step/self.N))\n",
        "            done = True\n",
        "        else:       # hold\n",
        "            if self.day_step == self.N:    # at maturity\n",
        "                reward = max(self.K-self.S1, 0.0) * np.exp(-self.r * self.T)\n",
        "                done = True\n",
        "            else: # move to tomorrow\n",
        "                reward = 0\n",
        "                # lnS1 - lnS0 = (r - 0.5*sigma^2)*t + sigma * Wt\n",
        "                self.S1 = self.S1 * np.exp((self.r - 0.5 * self.sigma**2) * (self.T/self.N) + self.sigma * np.sqrt(self.T/self.N) * np.random.normal())\n",
        "                self.day_step += 1\n",
        "                done = False\n",
        "\n",
        "        tao = 1.0-self.day_step/self.N        # time to maturity, in unit of years\n",
        "        return np.array([self.S1, tao]), reward, done, {}\n",
        "\n",
        "    def reset(self):\n",
        "        self.day_step = 0\n",
        "        self.S1 = self.S0\n",
        "        tao = 1.0-self.day_step/self.N        # time to maturity, in unit of years\n",
        "        return [self.S1, tao]\n",
        "    \n",
        "    def render(self):\n",
        "        \"\"\"\n",
        "        make video\n",
        "        \"\"\"\n",
        "        pass\n",
        "\n",
        "    def close(self):\n",
        "        pass"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8gy2JFxz-r8K",
        "colab_type": "text"
      },
      "source": [
        "To simulate stock prices, set policy to not exercise or hold"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "edCHLZhwPlws",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 330
        },
        "outputId": "9dc23049-65e4-4b74-85e9-bbce6cbedf6b"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "env = AmeriOptionEnv()\n",
        "s = env.reset()\n",
        "\n",
        "sim_prices = []\n",
        "sim_prices.append(s[0])\n",
        "for i in range(365):\n",
        "  action = 0\n",
        "  s_next, reward, done, info = env.step(action)\n",
        "  sim_prices.append(s_next[0])\n",
        "\n",
        "plt.xlabel('Date')\n",
        "plt.ylabel('Stock Price')\n",
        "plt.plot(sim_prices)"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n",
            "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f33f4a8a128>]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2dd5hkZZW431PdVV3VOc/05MgMMwMMMCQJEgQRVMSAuK64Liur67rGVTHrym/dXXV1F0VRESOIAoKLAkMcMgzDZCbn6ZnOoTpU/n5/3Hurq7ur43SFrjrv8/TTVd+9t+r07ap77slijEFRFEVRAFyZFkBRFEXJHlQpKIqiKHFUKSiKoihxVCkoiqIocVQpKIqiKHEKMy3AiVBbW2sWLFiQaTEURVGmFa+++mqrMaYu2bZprRQWLFjA+vXrMy2GoijKtEJEDo60Td1HiqIoShxVCoqiKEocVQqKoihKHFUKiqIoShxVCoqiKEocVQqKoihKHFUKiqIoShxVCoqSRzy4qZGuvnCmxVCyGFUKipIntPiD/Mtdr/HApqOZFkXJYlQpKEqe4A9YFkJ3v1oKysioUlCUPKEvFAWgJxjNsCRKNqNKQVHyhN5gZNBvRUmGKgVFyRN6Q5Yy6FGloIyCKgVFyRN6g477SJWCMjKqFBQlT+gLqftIGRtVCoqSJzgBZkcp3P/aEe5/7UgmRVKykGk9ZEdRlPHTFxwcU/jU7zcBcO3pczImk5J9qKWgKHlCT9x9pCmpysioUlCUPKFvHIHmu14+xK9fOJAegZSsJGVKQUTuEJFmEdmasPZfIrJDRDaLyP0iUpmw7WYR2SMiO0XkzamSS1HyFScltTcUiQedh3LzfVv4ygPbMMakUzQli0ilpXAncOWQtbXAKmPMqcAu4GYAEVkBXA+stI/5kYgUpFA2Rck7nACzMXC0oz++HggPdyftaupJm1xKdpEypWCMWQe0D1l71Bjj3KK8CDgRrmuAu40xQWPMfmAPcHaqZFOUfMRpcwFwqL0v/tgfGLAaXGL9fn5va9rkUrKLTMYU/h74q/14NnA4YdsRe20YInKTiKwXkfUtLS0pFlFRcofE+oQX97XFH3cHBhrkVRV7ANjW2J0+wZSsIiNKQUS+BESA3070WGPM7caYNcaYNXV1dVMvnKLkKL3BKAW2KfDTZ/bH1xO7pjpxh2Z/ML3CKVlD2pWCiPwd8Fbg/WYgmnUUmJuw2xx7TVGUKaI3FGFRbcmwdcd9FI0ZAuEYAM3dgbTKpmQPaVUKInIl8Dng7caYvoRNDwLXi0iRiCwElgIvp1M2Rcl1eoMRzllUzfNfuHTQuuM+SsxIalFLIW9JZUrqXcALwDIROSIiNwK3AmXAWhHZKCI/BjDGbAPuAbYDDwMfM8ZohY2iTBHRmKGrP0xVsYdZlT7+9c3LWD6zDIDufksZOIHourIi2npDhKOxjMmrZI6UtbkwxrwvyfLPR9n/FuCWVMmjKPlMV3+YmIHqEiuQ/LFLlvCh8xew4quP0B0IE47GeO1QJwALa0po8Qdp7QnSUOHLpNhKBtCKZkXJA9p7Q8CAUgDwuQsodAnd/WFueeh1PvKbVwFYUFsMQHO3upDyEW2Ipyh5QEefpRSclFMAEaHc56arP8wG20oAWGAHozUDKT9RS0FR8oBklgJAVbGbjr4Q7gKJry2scZSCZiDlI6oUFCUP6LCVQtUQpVBd4qG9N4S7YOBSMLvKiiN0JdQvKPmDKgVFyQPabfdRdXFypVAgA5ZCVbEHd4HEs5KU/EKVgqLkAe09IXzuAnyewX0mLaUQHtTqorSokHKvG39ALYV8RAPNipIHtPeFhsUTwFIKHX0hEgwFiosKKPMW0h1QSyEfUUtBUfKAjt4QVSXuYevVJUVEY2ZQBbOnwEW5Ty2FfEWVgqLkAe194UHpqA7VSRSFiFiWggaa8xJ1HylKHtDRG2JBTfGw9eqSovjjT1y2lDULqgAo97q1eC1PUUtBUfKAjt4RYgoJ1sNpcyu4cKnVjt6KKailkI+oUlCUHCcUieEPRoalowJUlw6snTI7PjLdzj7SQHM+ou4jRclx4i0uklgKsyq8fPzSJVx9agN1ZQOupDKvm75QlHA0NqiwTcl9VCkoSo4zUosLsILKn7li2bD1cp91aegJRJIqEyV30VsARclx4i0ukriPRqLMa2UlqQsp/1CloCg5jtPioqZ0/Eqh3GtZChpszj9UKShKjjMZS6HcZ1kK2hQv/1CloCg5TnuvdWGvLB5eqDYStaVW0FlnNecfqhQUJcdp6w1S7i2cUBZRfbmlFHSmQv6hSkFRcpwjHf3MrhpezTwaZUWFeN0urWrOQ1QpKEqOc6CtN2mLi9EQEerLvDqSMw9JmVIQkTtEpFlEtiasvUdEtolITETWJKwvEJF+Edlo//w4VXIpSj4RjRkOt/cx3x6xORHqy4rUfZSHpNJSuBO4csjaVuCdwLok++81xqy2fz6SQrkUJW9o7OwnHDXMn6ClAFBXVqSWQh6SMqVgjFkHtA9Ze90YszNV76koymAOtfcBTEop1JcV0aIxhbwjm2IKC0XkNRF5WkQuHGknEblJRNaLyPqWlpZ0yqco046ndjYDsLB2Eu6jci/+YIT+UHSqxVKymGxRCseAecaY04FPA78TkfJkOxpjbjfGrDHGrKmrq0urkIoyndje2M1Pn9nPu8+cQ0OFb8LHL59ZBsC63XrzlU9khVIwxgSNMW3241eBvcBJmZVKUaY3e1t6APjwhYsmdfwbT6pjRnkRv33p0FSKpWQ5WaEURKRORArsx4uApcC+zEqlKNMbJ0hcn9ASeyIUFri44bwFrNvVwvN7W6dSNCWLSWVK6l3AC8AyETkiIjeKyLUicgQ4D3hIRB6xd78I2CwiG4E/Ah8xxrQnf2VFUcZDsz+Au0Am1N5iKDdesJDZlT5ue2rvFEqmZDMpm6dgjHnfCJvuT7LvvcC9qZJFUfKRFn+QutIiRGTSr+F1F3D6vEq2NXZPoWRKNpMV7iNFUSbOzuN+Ht56fMTtLf4gdeXeE36furKieGM8Y8wJv56S3ahSUJRpypu/v46P/OZVYrHkF+rm7uCk4wmJ1JUV0ROMsOlwJwtv/gsv7ms74ddUshdVCooyDQlHY/HHjV39Sfdp6QkOmrs8WersNtqPbLOskh9pfCGnUaWgKNOQTYc744/3tvQO2x6KxGjvDU2JpVBvu6CcFNctRzrVjZTDqFJQlGnIxkSl0NwzbPu+VmttduXEi9aG4lgKW450AdDRF9aeSDmMKgVFmYa09ATxFLio8LnZ0zJcKTyxw2pvceHSE6/6d1xQjV0DHVM77LnPSu6RspRURVFSR4s/SG2phxkVXg60DncfPbmjmZWzyplZceLZR9UlHlwCMQMFLiEaM3T06uzmXEUtBUWZZgQjUSvdtKyImhIPHX3DL9A7jvlZM79qSt6vwCXMq7a6rJ7cYPVD6lRLIWdRpaAo04jOvhDLvvwwz+xupba0iAqfh+7+wUohFjP0hCJUFHum7H3ffeYcgPic52SKSMkNVCkoGSESjdHWo8HKiZIY4K0rK6LC5x52194bimCMNWd5qnj/OfOZV13MP128BIDOfrUUchVVCkpG+J8n9nDmtx5TxTBBOhPu0GtLi6gsdtMbinLHs/tptc9lTzACQKl36pRCVYmHdZ+7hMtXzMDrdg2SQ8ktVCkoGeHl/VZV7AMbGzMsyfQiMeunttRDhc9qdvfN/9vOR379KgA9AVspTKGlkEilz0NHr1oKuYoqBSUj1Ni57w9sPEpnX4jLvvvUoNx7JTmJriKXa3AH1PUHOwDw25ZC2RRaColUFrs1ppDDqFJQMkKr7Rvf2eTntcOd7G3p5bk92rN/LBIvxovrSin3DW6LHYnG4pZCqpRCVbFHs49yGFUKSkZw/N+BcIxndlnKIFllrjKYjt4QnkIXz33hUs5fUkvlEKWws8mPP+4+mvwchdGoKnHT2a+WQq6iSkHJCK09ofgweafR2t4kRVj5yqsHO9h8ZLg7raMvRFWxO96+omKIUmj2B+kJWhfsqQw0J1KplkJOoxXNStoJRqJ09Yd508kz2N/ay9FOq8vnvuYejDEnNBRmuhKOxvjifVv4xzcuZlall3fd9jwA5d5C5teUcPdN51JSVEhHX5iqhPqDyiG1CP5AJG4ppCym4HPT2RfO2/9VrqOWgpJ22nqsu8zVcyvia1XFbvzBSHyYSy4Qica49Ynd4wqgH2jt5Q+vHuHJHc38/pXD8fXuQIQtR7t47PUmwAo0JwaXy4dc+P2BcDwltcSTuphCJGbiAW0lt1CloKQdJ54ws2Kgg+e/vWMVABsOTf8MpFcOtPPhX63nWw+9znce3cWnf79xzGOcZnOtPUHW7WoZtv2l/dbI8o6+MNUlA9ZBoV1hXOCy7tgdS6HEUxBfm2ocpdSp/Y9yEnUfKWnHUQq1pR5+8aGzKHYXcPq8KkqLClm3u4UrV83MsISTp7MvxD/8cj1d/WF87gIA9rX2srvJz9IZZSMed8x2obX0BHn9mH/QttKiQl6yp51ZlsJgl9E9/3gec6t9XPAfT1qWQiCSsngCEHdfdfaHmEdxyt5HyQxqKShpx6mGrSr2cMmyes5ZVIOn0MUbFtfw9M6WCQ1wMcbwi+f205UFefMt/iCrv7mWLjszpz8c5YoVMwB4dox022O2pbCnuYfj3YFBd/kXL6tjb0sv3YEw3YHIsFjB2QuraajwUeYtxB+I0BOMpKxwDQYsBa1VyE1SphRE5A4RaRaRrQlr7xGRbSISE5E1Q/a/WUT2iMhOEXlzquRSMo9z0RyaY3/G/CqOdvbTG4qO+7X2tfbyjT9v5y9bj02pjJMh2YyBU2ZX4Cl0xS/6yXh+Tyv3v3YUgM32IJuzF1THty+ttyyMpq4AoUhsxJ5GjlLwByOUeVOTjgoDwW3NQMpNUmkp3AlcOWRtK/BOYF3iooisAK4HVtrH/EhEClIom5IBfrpuH/dtOEJ3vxWgHBokdXLuna6fu5v8REcYSu/gXJjas6DtQk+SwOuMci+zKrw0diafowzwNz97iUPtfYPWzlo4oBTmVluxlyP2a4xkBZQWufEHwnT1h1OWeQRWUgCgrS5ylHEpBRG5QEQ+ZD+uE5GFYx1jjFkHtA9Ze90YszPJ7tcAdxtjgsaY/cAe4OzxyKZMH+5+5RAPbGykqz9MiacgHiR1cCyHrv4wjZ39XPH9dazdfnzU13RcUdlwgXIqiROpKy+iocI3qlIYyuxKHwtriwc9BzjaYSuFEayAMm8h3YEInX2hQWmrU41TG7FudyvHR7GAlOnJmEpBRL4GfB642V5yA7+ZYjlmA4cTnh+x15LJc5OIrBeR9S0tw7M0lOylLxSlNxihOxAeVnQFAxeb410B9rf2Ygw0do5+0XFcUe1Z4MpIZinUlxXRUOkd5D666Vfr+fWLB2nxB4lEY8OOWVRXQpldjVziKaCm1LrAH41bCsmN6HLbfdTRG4rfzaeCwgIXnkIXT+xo5oN3vJyy91Eyw3gshWuBtwO9AMaYRmDkNIoUY4y53Rizxhizpq7uxOfPKumjN2gFQbv6w8PiCQDl9h3wh+58hY/+xur42TVGOwXHUsgK95FtKXzm8pNYPtP6itSXeZlV4aOpO0AkGuN4V4BHtzfxlT9t5axbHuMvWy1L6PNXLudf37wMgHnVxfHsoTKvO36ujjiWwgjtK8q8brr6QnQHIsMylKaaUMRSZkPdXsr0ZzxKIWSsdBADICIlKZDjKDA34fkce03JEYwxlqUQitA9glJItB667QvsWErB2Z4N7iOnmOuG8xYwo9xLgUuoLvEwq9JHzMBXHtjKhkMdg4754RN7AFhcV8LFy6ybnPedPS8eNyjzFsbPy9EO6wI8UrppmbcwXu+QSkshkeUNGbs/VFLEeJTCPSLyE6BSRD4MPAb8dIrleBC4XkSK7HjFUkDt0hzACRSHojEiMUNv0Gpxkcx9VO4bfrEbr1LYcrSLX794kA2HOnj9WPcUSD5xHEuhpKiABTXFzK3yUeASFtdZ91F3vXyYtdub8BS62PWtt3DdmjnsbLJqEmaUe1k5q4ID376aVbMr4lZTmbeQosICij0FY7qPEoPLVSWptRS+fPXJAPRPIFNMmR6MqRSMMd8B/gjcCywDvmqM+d+xjhORu4AXgGUickREbhSRa0XkCHAe8JCIPGK/xzbgHmA78DDwMWOMftqmOW09QVZ+7WG+9X/b6Qta/86eoGUpJFMKydIox3YfWRZCzMBX/rSVd/7oed7yg2emQPqJ4w+EKbYD6J998zJ+9+FzAauO4GOXLAaseoWVs8rxFLq4+tRZ8WPry4sGvVai+wgsK6qp2yr6G8195JBq99E/XLiI69bMSZqGq0xvxsxbs+/cnzHGrLWf+0RkgTHmwGjHGWPeN8Km+0fY/xbglrHkUaYPh9r7CIRj/OzZ/ZyzqAawfNFtvaH4nXAiydoyjNdSyAYSi8bKvO74RVpEWDO/GthLiz8Yr0E4d9FA2mlt6RClkOA+AkspOMHqkdxHM8u98cfpcB9ZcxWy5/wrU8N43Ed/ABJTJKL2mqKMSuIF44W9bfHHwUgsqaWQjDEthSxSCv7gyO0lEl1jFfYFu6iwIF4d7B6SnuspdFFU6BqkFByK3cndRytnlccfpzIl1aGi2E0wEiMQVqM+lxhPhUuhMSZuIxpjQiKS+k+cMu1JdC2sPzioZCVp/CAZY92JZkN7C4eeQGTEauNEyyhxMM5Tn72YthGC5DdesJBzbQvLUR6lRYW4Rmh0t6iudOA90mApVPqsy0BHX4iGhOaGyvRmPN/MFhF5uzHmQQARuQbQuYnKmDi9cU6aURpv3+AwXkuhu3/0vv1d/WHedcYcTp9Xyf9tbuTFfZbyCUdjw+6+U03PqJZCor9/sO9/JP//565cHn/s3PmXjBBkhsHut1T2PnKId0vtC6tSyCHG8635CPBFETkkIoexCtn+MbViKblAR28IlxC/201kZoU3yRHDCUVjBMLDC7zAmlfQ2R+mocLL3547n797wwLqyyzffCZ83T2BkRvRlZ9gEHiNHYdwgs0j4aS1pmP4TaJSUHKHMW8njDF7gXNFpNR+roN0lXHRYbd5XlJfOmzbnMrkLZf/+72nseO4n/eumcsTO5r51kOv09kfwucZfid6vDtANGaYXWVtu3JVA6Go4V/ueo2vP7iNr719BfVlw5XPgdZemroD8eD3VGEFmpNbQF63C3eBEI6aYXOVx8Oly+vHtd/PblhDZIx+UVOF4z7Sxni5xYhKQUT+1hjzGxH59JB1AIwx30uxbMo0p7MvPGiecCIjWQrXnj4n/njHcX/8dYa6J453BXhiRzMAc6oGtjlZNw9tOUaR28X3rls97D2+9KctHGzr49nPXzrBv2h0ugMjN6ITESp8blp7hs9DGA/VJR5mlnuZVzP6/ILCAheFaWolqS20c5PRLAWncllLFpVJ0WE3ZptdNVwpeArH9lw6rqDj3QFObigftO2t//tsfFjPnKqBC2Vi1o0riQulozfEi/vap7yLaEdvCH8gklQBOpR7HaUwuSDws5+/JGXT1CZDbWkR7gLhcIe2usglRvxmGGN+Yrev7jbG/HcaZVJyhPbeEHOqipk1yoVyNBxl4nQHTcRRCAANCVZH4gV3qOLp6g+zdnsT0ZiZ0krcSDTGlqNWID2Zq8yhzHYbTcZ9BAzrKptpPIUuFteVsiNDFeRKahj1dskYExWR9wGqFJQJ09kX5pTZ7qSFauOhvsxLoUvi7R1GwpuQt59oKbTZiuOVA+38eVMjv3rhYHxbMBIjGjNTcuf9uT9u5j57SM5oSsGZH5Gs79N0ZfnMsvj8aCU3GI8N/ZyI3Ar8HrtTKoAxZkPKpFKmPbGYsdxHSXrwOBkyY1HgEhoqvUkthapid1JfdrFnQEG09lgB0I//7jWOdw+0ri4qdMWLrkqmIHXTUQjA6O4jnxufu2CQEpvuLG8o508bG+nqC8eL8pTpzXjs0dVYE9G+CXzX/vlOKoVSpj8H2noJRmLxZnAOL9x8KT/+2zPH/TqzK33DLIVozNDZH8ZdIHz7nacM2iYirP3URVy6vJ4Wv2UpBCJRako8fO5KqzX11ac0ANZ8h6lgQULwd6TCMoDlM8pyrquo0yJ8x3F1IeUK47lNeo8xRovVlAnhFKudOqcSgD997HyauwMTLnKaXVnMc0OG3nf0hTAGvnz1Cq4/e96wY5bOKGNhbQkv7msjEo3R1R/m45cu5Z8uXsJ7zpzLUzubue+1o1PWnqGrP0yhS/judaeNut/HL1vKxy9bOiXvmS04CQA7jvunPMVXyQyjpaS+DbgDCItIDLjOGPN82iRTpjWbjnTidbtYavvYV8+tnNTrzK7y0eQPDKpQdoLMQ5vIJVJbWkRfKMqRjn6MgTo7k6murIhij/WxnwpLIRCO0tEX5rNXnMQ1q5MOC8xp6suKqCp2q6WQQ4zmProFuNAYMwt4F/Dv6RFJyQU2Hu5k1ayKE86YmVPpwxgGzQJus2MFzpjKZNTa25yLVV3Cvj6PJVP/JC2Fj/1uA5/74yYAmu0K4xnl46vQzjVEhGUzy+I1Jcr0Z7RvbMQYswPAGPMSWq+gjJOuvjCbj3QlbW8xUZy01CMJweYBS2FkpbCw1opl/GH9EXvfAavC53YsheEzlcfDQ5uPcY/9uk4Ae7xtO3KR5TPL2XncTyxNldRKahktplA/pJp50HOtaFYcwtEYLpF4eucze1qIxgyXLD/xGdpONk9isNnJKqopGdl9dOb8Ks5ZWM3jdtWz4z4C8NkZSlMRU4grhTy1FABWNJTTF4qyv62XxXUjp+Qq04PRLIWfYlkHzs/Q54oCwClff4T3/fRFAB7ddpyvP7idymI3q+dWnfBrN1RaF9vEtNS2niCFLhm106qI8JGLF8efJ1oKTtrqZGIK1rhyi3A0RrOtFOrzWCmcPs+KF2042DHGnsp0YLSK5m+kUxBlehKNGQLhGC/vb2dPs59//t1rLKkv5atvWzElhWFFhQXUlxVxtHOglUJbT4jqEs+o6Z8A5y+ujT9OrEfwuSevFBI7tjb7g3T2hSlwSbwwLR9ZXFdKubeQDYc6ec+auZkWRzlBsqtuXslKYjHD1qNdSbcdbIvXM/LKgQ5C0Rg/ev8ZUxJPcJhd5eNPGxtpsu/K23qD1IySeeTgKXRx7qLqYe2sHfdRiz/I9x/bRbM/MG5/eGf/QEfQ410Buux50+loVZ2tuFzC6fOq1FLIEVQpKGPy8LbjvPV/n2VX0/AMk8S1Q+19FLhkUNfSqWBhTQmhSIy/v/MVjDG09IRGDTIn8psbz2HDVy4ftOZYCv/1yE6+/9huLv/eOhZ98S9jxhicmgeHd932PM/taR33wKBcZtXscva09BCOJp99oUwfxlQKIjLslkxEqpPtO2SfO0SkWUS2Jh4nImtFZLf9u8pev1hEukRko/3z1Yn+IUrqeO2QdQe4vXF4LvqupoHxGpsOdzKz3Dvljdu+ePXJvOuMOWxr7Gbd7lbaeoKj1igkUljgGtYYzzekzYRzoT+QYPUMZcuRLlZ87RE2HOwctL6vtTenehlNlgU1JURjZlCWmDI9Gc+39z4RiX/qRaQBWDuO4+4Erhyy9gXgcWPMUuBx+7nDM8aY1fbPN8fx+kqa2HrUUgZ7mofPV9qdsLb+YAezKqc+4FpbWsS/v/MUZpZ7ue2pPbT1hKhJ0lNpvCTGItbMHwiG72sZWSm8cqCdUCTGI9uOD9s22a6nuYSTAnygdeRzqEwPxqMU/gTcIyIFIrIAeAS4eayDjDHrgKHtE68Bfmk//iXwjnFLqmQEYwzbGq14wu7m4e6j/a098bkHoUhs1IZwJ4Kn0MU/XLiQF/e10x+OjiumMB4+c8Uy3msHR/e1jDxUcI+9zWm58f33ro5bIOo+ggW2UtjX2sv3Ht2pymEaM6ZSMMb8FHgMSzn8GfiIMebRSb7fDGPMMfvxcWBGwrbzRGSTiPxVRFaO9AIicpOIrBeR9S0tLZMUQxkvRzv76Q5EcMlwS8EYw/6WXt6weCCoPNnZCeMhsc/ReGMKY3H6vEr+492n0lDhHdVScP52Z9Tlm1bMYIXd90eVAtSUeCgtKuS5Pa38zxN7BnWOVaYXIyoFEfm08wN4gXnARqx5zZ8e6bjxYqyEbyflYwMw3xhzGvC/WApopONuN8asMcasqas78eIoZXScTqMrZpVzoK2PnuBAFXBLT5DeUJTVcyvjF8aGFCqF0qLCeBFa9Qm4jxJx2lgvqith7yh3t3uHKMQST0FcMU12klouISIsqC3mmd3WjdpoVpeS3YxmKSQWqpUC9wF7OLHitSY7JuHEJpoBjDHdxpge+/FfALeI1I78Mkq6cIKwV53SQDRmeGbXgHV2oNWqHVhYV8qjn7qIvz9/IW9ZNTOl8vz9+QsBawDPVHLmvCo2H+lky5HhqbedfSHaekO858yB+dEigs9urKeWgsWSulLCUes+bzSra6oJRqJT1vFWSX/x2oPAB4Fv278fABCRmUCTMcaIyNlYyqotBe+vTBBHKVy2fAa3r9vH2tebeIs9j2B/q3U3uLCmhBnlXr76thUpl+cjb1zEpcvrWTbzxIrqH//MG/EkZEn9w0WL+O1Lh/jRU3u4bci8B6etxoUn1fH21bM41G4pQ3eBFbCeikE9ucCymeVAIwD7W3uJxcyYBYZTwef/uJnuQIQ7/u6slL9XPjDmp1lE1mLNVOi0n1cBdxtj3jzGcXcBFwO1InIE+BqWMrhHRG4EDgLX2bu/G/ioiESAfuB6k9hPQMkYjlKoLvFw/uJa1h8YKFA61N5HoUtSknE0Ek5XzhNlaI+ecq+bM+ZXsTeJ28NxmZUVFXLh0gGXZZEdaI5oIzhgYOAOWB1oj3T0My9hAFGq2H6sm96gWgpTxXiyj+ochQBgjOkA6sc6yBjzPmNMgzHGbYyZY4z5uTGmzRhzmTFmqTHmTcaYdnvfW40xK40xpxljztW5DdlDlz3yssLnZlallxZ/MN7/p7EzwMyKqa9LyBSzK30c7ehn6P1IT8BSCqVDWkERuv8AACAASURBVFk4LqyhdQ/5ylBlfeUP1k26E+1ovOOHz/Fv/7c9/vxYZ2DQ51I5McbzbY6KSDztQ0TmMxAgVnKczv4wxZ4CPIUu6sqK6A9H6bV7Bh3t7E9ptlG6mVPlozcUHVS1DOAPWM+Htsv46MWL+epbV/CO1bPSJmM202C3D59d6eO6NXPoC0UHzcGYCowxbDzcyc+f3Q9Y/xt/MEJoSLW5MnnG4wz9EvCsiDwNCHAhcFNKpVKyBqe3Dwy0n27xByktKqSxs5+zFoxZ3D5tcGosjnT0U1k8kN3kd9xHQywFr7uAv79gYfoEzHJEhIc/eSE1JUVsPdrFPeuP0DnFF+rehCaG3YHwIKXT7A8O+r8pk2M8dQoPA2cAvwfuBs40xjySasGU7KCzL0EplFp3gi3+INGY4XhXIK3xhFTjDPRJnN0AA+6jsiLNMhqL5TPLqSsrosJO053qu/dWO0Ua4LVDnTQm/K+cKXjKiTFeZ/AbsILGFwPnpkoYJfvoTrAUasusu7AWf5AWf5BIzOSU+yg+0GdI/x4n0FxSpLGD8eJ8Zrr6wgTCUb76wFZ2TsHIzpaeRKXQwbFBlsLUuqrylfE0xPs28Algu/3zCRH5f6kWTMkOBrmPSh33USB+N51LSqG6xEOJp4D9Q4rY/IEwPndBzgTU04HTD6qrP8zXHtjGr144yF0vHzrh1020FLY3dnMs0VLwq6UwFYwnpnAVsNoYEwMQkV8CrwFfTKVgSnbQ2R/itOIKAKqKPRS4hJaeYLxl9rzq1KccpgsRay7A+iFzAXqCkWHxBGV0nM6xRzv7uXeDNc+6oy802iHjwpnPfe6iarY1dlPqLaShwktXf1jdR1PEeG99KhMeV6RCECU7SbQUXC6httRDiz/IkzuamVXhZZHdCC1XOHthNTuOd8dTcQH8gciwdFRldNwFLkqLCnl463EiMUOhSzjQ1jf2gWPQ0hNCBC5YUsvRzn62Hu1iXnUxJ80oY93uFk1LnQLGoxT+HXhNRO60rYRXAXUf5QHhaIxAOEaZdyDAWltaRGNngGf3tHLJ8vqcmzh29sJqjIH1Bwca/PoDEcq0annCVPjcHGrvo9hTwDvPmD1oSl8yntvTyvN7Wke8sL96sIP/eXw35V43p8yx7lN3NfUwt7qYG86bz57mHp7epU0yT5TxZB/dhRVcvg+4FzjPGHN3qgVTMk9vPMA6cEGsKvaw4VAHfaHolI7czBZOmW0ZwjsSgqI9QbUUJoNjYZ61oJql9WV09oUHWWBgzfh+amczz+9p5f0/e4m/+dlLPLTl2LDXMsbw8d9tACzr9bQ5Aw6LedXFvPXUWdSXFXHHcwdS9wflCeMJND9ujDlmjHnQ/jkuIo+nQzglszhZN6UJWTdVJZ74wPsZ5bmTjupQUlTIzPLBbbR7AhFNR50EhXZvqDPmVcXbXQydbvfnTY383S9e4W9+9lJ8bdPhwdPtAF7Y20ZjV4BzF1Xzg+tXU2nHtwDmVvvwFLq44bz5rNvVkrRViTJ+Rmud7bXHbtaKSJU9SrPaHrQzO10CKpnD6Scz2FIYuDhOVfvqbGNhbQn7WgcuLP5AWC2FSeAM2lk9r5KTZlgtMF4/Nnika0uSjKHGzuGppY/vaMZT6OLOD53NNauty88Mu5hydqWlcC5dbo1n2TUFqa/5zGiWwj9ixQ+W27+dnweAW1MvmpJpekPJ3UcOUzXoJttYVFcyKC3VH4wMa3GhjE23XfS3ek4lC2qKqSx289qhwVaAPzi4N9IbFtcMKx4EeGl/G6fPrYzPvwD41yuXAbCk3mpuWFVi3bBMdRV1vjFa6+wfAD8QkY8bY/43jTIpWUJv3H003FIocAnl3tx0qSysLaGzL0x7b4iSogL8gUjOWkWp5Hf/cA4v7GuLVzevnlvJxiGuoZ6ApXAvO7meSNRQUlTAUzsHB4u7A2G2N3bzz5cuHbR+7elzuPb0gRkXlT7rf9TZp0rhRBhRKYjIWcBhRyGIyA3Au7BaXn/d6XCq5C7xQLMnQSnYF8fqEk9aeuVnAsfVselwJyfZnT+dOdTK+HnDklresGRgVtbpc6t4etcu/IFwPKOt17bCfnD96QD84LHdNPuDBCNRigotq2DT4U5iBs4eo8+W1+3CU+iis//E6yHymdHcRz8BQgAichHWLIRfAV3A7akXTck0PfGYQkKg2XYf1eTwnfM5i6qpLHZz32tH4z7vOlUKJ8zKWeUYY6WROgzN7HJ6aSU2utt61IpDrJpdPurriwiVPvewDCdlYoymFAoSrIH3ArcbY+41xnwFWJJ60ZRMkywl1XGj1ORoPAGgqLCAt506i0e3HY/PGlalcOI48xacangYHq+J959KiCtsa+xidqVvXB1QK4vd6j46QUZVCiLi/LcuA55I2KZRtxwlFImxzi4A6kkSU3CG1NeU5PZF8l1nziEYiXHn8wcAVQpTwexKH8WegkGN8XoC4UGfr/rygfbsDtsbu1k5a3QrwaHS51H30QkymlK4C3haRB7AGpH5DICILMFyISnTFGMMn/79Rj5x92vDUgI/+fvXuOGOl9l53E9fKEKBS+JjJ2HAUsj1wOtpcypYXFfC5iPWR722VJXCieJyCUtnlA1WCkMshdrSAaXQ2Rdif2sv+9t6WTlrfN11KiZhKXz9wW3cs/7whI7JZUZUCsaYW4DPAHcCFyTMTHYBH0+9aKmlLxThwU2NmRYjI/SFotz32lEe2NjIUzub4+tdfWH+suU4YH0pe4NRSjwFg1pZ+NwFXHXKTC46qXbY6+YSIhLPhwerl49y4iybUTrIfdQzpK9Uhc+Nu0Bo7Qnx9luf45LvPIUxcOE4P28VPjfdE0hJjcUMd718iD/n6bUgGaN+0o0xLxpj7jfG9Cas7TLGbEi9aKnlc3/czL/c9dqU9HifbiR2q2ztGXi84/hAYVFbb3DYXRxYF8sfvf/MeKFQLnPFytz/G9PNorpS2npD8eE7Q2MKIkJtaRGHO/o41D7QQG/1nMphr5WMSp97zDqFUCTG3S8fIhCOcqw7QDASY2+zVkE75G1s4JUDVgy9PxwdY8/co6N34EuT6D5KDO619YToDUYGBZnzjWV2amqO9fzLKAtqrK66+1p6+OOrR6xmg0OqxWtLi3hqx4AF+6HzF4w7/bmy2E1fKDoopXUo3127k588vQ+XSHzaXmNXgL5QhGJP/n7eHVJ6BkTkDuCtQLMxZpW9Vo012nMBcAC4zhjTIZaP4gdY8xv6gL9LpUXSZPdez8dh34mWQuIkq8SJY+29IXryXCmICI99+o141HU0ZSy0W62v3d7Eb1+yhu4MtUbryorYcrQLd4Gw4SuXT6iavMLOUOrqC1NfPlwphKMxfv3CQQB2NvkHWSP7WnpZNVsnA6T6034ncOWQtS8AjxtjlgKP288B3gIstX9uAm5LlVCJrXnzWSnUlHhoSRhheKSjn9rSImpLPdz9yiGe2d1KYY4WqI2XJfWl8WZuyokz3z6Xf9484MMfeuPhtE85fW4VZV73hNqzl9tWh9NiYyjtvaF4Q8efP7ufW5/cE9+2r3X01t75QkqVgjFmHTC08vka4Jf2418C70hY/5WxeBGoFJGGVMiVONc1L5VCr6UUTppRNsx9NLvKR3WJJx5r8Hl0LrEydXjdBTRUeDncPmCVJma3wUAG0nmLJ96a3XFF9QRHVgoAifc6js453H7iQ4BygUzYxTOMMU7D9OOAE82bDSTmhR0hSTdWEblJRNaLyPqWlskN1GjtCcbvRiaSqZArtPeFEbHugh2lEIrE2N3sZ06lL161vGp2Od+8ZlUmRVVykOV2EZvD0Lt6Rymcv2TiGW5O+4yeESwF54bIafv+L5ctZe2nLqLEU0Bbj9Y3QIYDzcYYIyITmp9njLkdu83GmjVrJjV779Q5laz/8uUs+/Jf89JS6OwLUeFzM7PCS3cgQiAc5QM/f4mm7iBzqn3xnvcfOHd+3AesKFPFv71jFV+6fyuXLKvjSEc/162ZM2j7W06ZSVd/mDPmjS/jKBEn/uAPJP9et9uu02+8fSVrtzfxTxcvxusuoKrEMyUzpHOBTCiFJhFpMMYcs91DTprBUWBuwn5z7LWUUZGnfVLae0NUFXuos+/IjnT088qBDmpKPHzwvAXc+6o1aH1FgwbdlKlnTlUxv/z7s0fc3lDh41OXnzSp144rhRHcR46lsHpeJVesnBlfrynx0NarSgEy4z56EPig/fiDWPMZnPUbxOJcoCvBzZQSKnzuvLQUOvpCVBW74+l4j263CtZuuXYVsyp98S/L0hmlGZNRUSZD+Rjuo3Y7HbtqSB+lqhJPXGHkO6lOSb0LuBhretsR4GtY3VbvEZEbsdpwX2fv/hesdNQ9WCmpH0qlbADl+aoUesM0VHhZag8nuX+DZZA56XjfePtKPvWmkwYNNFGU6YDT0dc/UkyhL0SZt3BYhXp1iYfdTVrABilWCsaY942w6bIk+xrgY6mUZygVPvegFr35Qld/mOUNZdSVFVHhc7O7uYfKYne8Q6W7wKUN4JRpSWGBC5+7gJ7gCDGF3lDSvl3VxZ54ZtJYvHqwg9PnVubsPJG8rsrJV/dRd3+YCp+V/z2v2sobv3Bp3YTywRUlWynzFuIPRNhypIvGIaM9LddpEqVQ6qE/HKU/NHqHgy1HunjXbc/z2OtNUypzNpH3SuFoZz/ffXRnpkVJG9GYwR+MxH2vTpuP68+aO9phijJtKPUW4g9GeNutz3LRfz45aNtIloIzNKp9jAwkpz/Yb186xK1P7J4iibOLvFYK7z1rLkWFLn794sFBVc7jJRqbVEZs2jHGsHZ7E6FILJ6qV+GzlML/u/YU3nf2PM5bNPFCIUXJRsqKCmmx29hE7O9ofyjKp3+/kW2N3XHrOBHHemgfpVahsy/E9mOWUnh6VwvfeXQX4WhsqsXPOHmtFE5uKOerb1tBZ194UIXlePjpun0s/uJf4tPJspl1u1v58K/W88Mn99Ddb8lbbiuFsxdW8+/vPCVn/aNK/lHmdfP6sYGOvwdae7nuJy9w/8ajfPTixXzmiuHprs4kwdEshdXfXMsvnjswaC0XaxvyWikAnGa35N10pHNCx9329F4A7nr50JTLNNUcsHu6NPsD8RiKYykoSq5RWlQ4qE7hPx/ZwZajXdz2/jP4/JXL41XPiTj9l0a6yYuN4BUYb3B6OpH3SmHZzDI8hS42T1ApeO1+LfdtSGl93ZTgtPIo87rptt1H5d787X6q5DalQz7bT+1socAlXL5i5ghHQLHbOqZvhEBza28w6XoutsbIe6XgLnCxqLaE/a3jb4bV0Rui0U5lPd6d/Smtx2wZI1EzYCkUq6Wg5CaVCVZwhc+arzCjrIiCUVykTuPH/lByS+FYp/UdWlJfyl8/cSFrP3URQE5WQee9UgCYWeHlePf4YwqOv/LshdW094YIRrJ7UI/T/bGzPxS3GsqTmNCKkgu8NyGTbm61VXszs8I76jHFtlIYyVI41mVdH77/3tWc3FAez2Bq60luQUxnVCkADRXeCRWxbT5qDXO/ZFk9AM3d2f3BcJRCV184bimUa0xByVGWzijjgY+dz70ffUM802gspeBzj64UjtqWwiy7wLOy2INLNKaQs8ws99HaM/47/o2HOplfU8zyBqsFcLM/e11IoUgsPmazqz9MdyBMgUso0TkJSg5z2txKzpxfxdwqWymU+0bd3+USigpdBIaM5339WDd/89MX2dfSQ1Ghiyrb7VrgEqqKc7OJnioFoKHSuosY7x3/xsOdrJ5byYwy67imLLYUtjV2EY5amROd/WG6+yOUewu1elnJC+bELYWx27YUewqGWQrP7m7l+b1tPLWzhVmVvkHfm+oSj7qPcpUG27Q8Ng4X0r6WHo53B1g9tzJukmZz/6RXD3YA8KaT69nT3MOvXzyo6ahK3jC3yokpjG4pABR7CocphUY7lnC0s585VYNfo6bUo9lHuYqjFHYc7+YXz+0fsbo5Eo1x4y/XU+Yt5E0nz6Cq2I27QGjKYvfRKwfamVddzNIZA9OuJturXlGmG2ctqOb6s+ZywTimuPk8BfSHB7KP1m5vYtvRgSK4oQOnFtaWsLPJP206G4wXVQoM3EV89YFtfOPP29l4OHnNwuvH/Oxv7eXrb1vJ3OpiRISGCh+H2vo40NrLfRuO0DdCSlum2NbYzWlzK+PZRlef0sA1q4dNOVWUnKSkqJBvv+vUpP2OhpLoPurqD/PhX63n5QMDI+YX1AxWCucuqsEfiLC9sZtU0OwP8Jctx0YsnEsVqhSwKiAX1Az0Q2kaofZg/UHrA5I4UPychdX8detxLv3uU3z6nk1c/r11vLC3DYBdTX6u+8kL/PDJPZPqrXSiRKIxjnUFmF9dzHHbDD65oWyMoxQlP/G5B5RCc5JrwMK64UoB4MV9bSmR53uP7uKffruBL/1pS0pefyRUKdicMa8q/vhgW/JCtvUHO5hV4Y2npQFcbKelxgzc/oEzKXAJn7t3EwB/3XKcl/e381+P7OSHT+5JofTJOd4dIBozzKny8b5z5jGnysd71mg3VEVJRrGnIN46O1nyyMIhlsKMci/LZ5Zx18uHUlKr5HgsntuTGqUzEqoUbE5PGBJ+YASlsOVIF6cnKA+AC5ZavsoPnDufK1bO5Ibz5nO4vZ+m7gBbjnayuK6EC5fWcvcrh1Mn/Agc6bCsgzlVxSyfWc6zn7+UGeWj52srSr5iBZoj/Nv/bedvf/7SoG2eQtewQDPAzVedzL7WXn7y9L74mjGGnz2zL95zbDL4A2F2NfkBqztrOtEGODbnL6nFU+giFImxu8lPbzASb5Ll0OIPxoPSDhU+Nxu+cnm8tH7NgmrAyvrZfKSLC5bUMqfKx3N7WglFYngK06eHD9nKLdmHWVGUwfjsmML9rw30M7twaS2fv3I5gXCUwoLh3903nlTHW09t4NYn93Dt6bOZW13MvRuO8q2HXmfzkS7+532nT0qWTYe7iBk4c34Vrx7sIBKNJX3/VKCWgs2iulK2f+PNXLlyJusPdnDND5+jrSfIP/56Pa8f6+b5va30h6PUlA7Pd64u8cRbT69oKKeo0MVDm4/R7A9yypwK5teUEDNwpGP8/ZVOlN+8eJDP3bsZGKjDUBRlZHzuAo51BQZVKf/6xnNYNbsifrOXjM9esYxQJMZTu1oA+IndQbk/PHmX0v42y8o4e6H1vt0jzJxOBWopJFBY4OLqUxt4eNtx9jT38Nk/bOLJnS28dqiTZr/lY3T6ro+Ep9DFmgVVPLTlGABr5lcTilofjoNtfSyqK03tH2Fzz/oBd1VRoVYvK8pYFE+yyn9+TTEVPjfbG632N8614mDb5N1HLd0BRGCJfb3o6Es+MS4VqKUwhLedNosD376aZTPKeHKnpfmdfzJA7RhKAeDik6zgc11ZEStnlTPfDlCdyIdkovTYfeG/ec3KtL2nokxnnE6p4/mOJyIirJxVzrbGbmIxE29Pf6Ctb9I1DM3+IDUlRfGb0M6+9M2Sz4hSEJFPiMhWEdkmIp+0174uIkdFZKP9c1UmZHNYMiP5HX1Nydjl8hcvqwPgkmV1uFxCTYmH0qLCEQPYU00gHOVAay//cukSbjhvQVreU1GmO46lcOly66YuMU19LFbOKmfHcT+d/WGMgcV1JYQiMRo7JzbR0aHZH6S+rCg+JrSrP33B5rS7j0RkFfBh4GwgBDwsIv9nb/5vY8x30i1TMhYPqV50GMt9BFbP9S9etZzLTp4BWHcSMyu8aWuct7uph5iB5Q3laXk/RckFfB7rcviGxbV84k0nUeoZ/+Vx2cxyQpEY22wX0uq5Vext6WVfay9zk8yEHotmf4C6siIq7QZ8Hb25bSmcDLxkjOkzxkSAp4F3ZkCOURlaqOJQmyTQPBQR4aaLFrM4IX5Q5i3En6Zg0RM7mgE4ZXZFWt5PUXKBpfWl1JZ6OH9JLbMrfRMaROXcLDregLMWWKnrO45Nrtq5uduyFCp9tvuoP7eVwlbgQhGpEZFi4CrAqaj6ZxHZLCJ3iEhVsoNF5CYRWS8i61taWlIm5MLa5O4jr3tywajSosK4nz+VNPsD3PHcft508oxJ3aEoSr5y7qIa1n/5curKxr7xG4rj5jlo1ybMqylmVoWXHz+9ly/eP7GK5GjM0NoTpL68iDJvIS5Jb61C2pWCMeZ14D+AR4GHgY1AFLgNWAysBo4B3x3h+NuNMWuMMWvq6upSJueiuhLcBTJlEf90WArGGD7ws5cJR2N85gpteqco6cKZs3DQHmhV4XOzYlY5HX1hfvfSIUKR2Lhfq603SMxAfZkXl0uo8LkHBZo7+0LxSXCpICOBZmPMz40xZxpjLgI6gF3GmCZjTNQYEwN+ihVzyBjlXjcP/vMF3PKOVXjdLq4+pYGrT2mY9OuVFbnpSbFS8Acj7Gzy87FLlnCyxhMUJW04bh6nYLTC56Y0ofh1Inf6zms4rflrSovig7LaeoKs/uZaLv6vp1LWKC8jdQoiUm+MaRaReVjxhHNFpMEYc8ze5VosN1NGObmhnOUzy3jjssspnkDQKRml3kL8gdT6BTvsopv6SZi/iqJMHsfNc7Ddch9V+Nzc8IYF/GljIwDtfSHqx9li5uldLRS4hHPswrXzFtVwz/rDrN3exNO7rHhhMBLjtcOdnDk/qZf9hMhUncK9IrId+DPwMWNMJ/CfIrJFRDYDlwCfypBsgxCRE1YIYMUUekPRlPZe77BNzHQVuSiKYuFyCZXFHgLhGC6BEk8hZ8yr4q4PnwtMbJbzY683c+b8KirtOMWlJ9cTjMT48K/W85sXD3H+kho8BS7+suXYGK80OTJiKRhjLkyy9oFMyJIuyrzWqe4NReKzDaaaDttEdT5MiqKkj8piN+29Icp97njbG+cGbbxKobGzn9ePdXPzW5bH185bVMOiuhL2tVhWyHVr5rK0vozZlanpaaZtLtKEoxT8gRQqBfuDp5aCoqQfpylm4rjbqhKnzmBspXC4vY8fPWX1Tbrs5Pr4utddwOOffiNHOvq5fd0+3rxyZkoHZalSSBOlRdaHI5XBZsd9VDWB/GpFUaYGJy01sT29s9Y+juKzz/5hEy/tb8frdg2qcQLLjT23uph/e8eqKZQ4Odr7KE0MWAqpCzZ39oVwCSmzRBRFGRmn59EVK2bE19wFLsq9hXHX7mi09Fg91r7y1hWISGqEHAeqFNJEqaMUpqiA7Z9++yrff2zXoLWOvhCVxQNtvBVFSR/hqJVE8pYhqevVJZ5xxRSC4RjvPGM27z9nfkrkGy/qPkoTZXbO8lS4j4wxPL2zhRZ/kE++aaBIraM3HO+VoihKevnB9avZeLhzWAC4ahxKwRirink8bXRSjSqFNFFmu3Smoqq5oy9MbyjK4fb+IeshqjXzSFEywvyaknib/ETqSoviozVHoicYIRiJTbhtdypQ91GaKJ2imMKrBzv4+F0bAGjyBwYNDO/oC2s6qqJkGecvqeVAWx97mntG3Ke1x7IkssFSUKWQJko8BXjdLloSBvYYY/jqA1vZcKhj3K/zrtue57k9bfbxcLRjwFpo8QeoK1OloCjZxBUrrcDzI9uOj7hPW48z2VGVQt4gIsyq9NGY0MiqqTvIr144yDt/9Py4XsP54CRy2FYKoUiM1p7QoHQ4RVEyT0OFjzXzq/jjq0cG9Sv6ziM7+c2LBwFotb/b6j7KM2ZX+gbd2e9rGTAnx9NW+7m9bcPWDttdGZ10tpmqFBQl63j/ufPY39rLbU/vJRyNEYnGuPXJPXz5T1vZ1thFSxa5jzTQnEZmV/p4/dhAwGlv68DM5nW7WrhqjC6sTV3W5LYbzptPMBzjwU2NbGvs4j8e3sGL+yyFMaNClYKiZBtvWdXA9x/bzX89spOl9aWDAtK3PPQ6Gw93IpId3QhUKaSR2ZU+WnuCBMJRvO4C9rX04C4QClzCy/vbx1QKzf4AXreLb7x9JSLC/tZe7nr58KB9ZpSpUlCUbMPrLuAPHzmPs295nOPdgXgW4vKZZTy/t42iQhdff9tK3AWZd95kXoI8YnaVlb/sDPPe39rL0voyzpxfxcv728c8vsUfpK6sKF7teNrc4eM2Z6qloChZSU1JES6xvsdbG7vwul3ceMFCAC46qY4PvmFBZgW0UUshjcyyi1p+8vQ+djb5OdbVzzkLrQ6IP3h8N609QUqLCkcc+dnsD1KX4HM8aUYZAGfOr+LVg1YGk/Y9UpTspMAl1JQW0doTZG9LLysaynnrqbPY1eTnwxctyrR4cdRSSCMLbD/i79cfZuPhTpq6g1y4tJYz5lVhDKz51mOc/s21Ix7f4g9Sn+AeunR5PStnlfPNa1bG1zLZM0VRlNGpKy2iqTvI9sZuVs2uwOcp4EtXrxj0vc40aimkkRnlRZR4CugNWQVnLoHLTp4xaPBOfziKMSbpxb2lJ8i5i2riz2tKi3joX6zRFH/4yHnjas+rKErmqCsrYv2BdnqCEVbNGu7+zQZUKaQREWFxfSmbj3QB8I7Vs+PZBnVlRfHCts6+MFVDshCCkSidfWHqRhi1edaC6hRKrijKVFBXVkS3HWReOTs756ir+yjNLKq1XEhff9sKvvfe1fH1FQ0DH5DDHX3DjnPK4EdSCoqiZD/O99dT6GJpfVmGpUmOKoU04wzPWFw/eIhG4gDuQ+3DlcJBu6YhVSP4FEVJH5efPANPYXZefrNTqhzmDUtqaKjwsnKIP/Ef37iIJz7zRoBh3U8Bthy1XE6rZmenH1JRlLF540l1AHzyTUszLMnIZCSmICKfAD4MCPBTY8z3RaQa+D2wADgAXGeMGX+nuGnCmfOreeHmy4atFxUWsKiulOoSzyBL4TuP7OR4d4BAOMrsSl9WVDwqijI5zl1Uw/5/vyqrswTTbimIyCoshXA2cBrwVhFZAnwBeNwYsxR43H6ed8yp8sWLyw7wnwAACF5JREFU22Ixw10vH+KpnS1sPdrFqiwNTCmKMn6yWSFAZtxHJwMvGWP6jDER4GngncA1wC/tfX4JvCMDsmWcGeVemrqtHkfbGrtp6w3R2hPkQFtf1qawKYqSO2RCKWwFLhSRGhEpBq4C5gIzjDHH7H2OAzOSHSwiN4nIehFZ39LSkh6J00hDhZdjduO7p3c1D9q2qK402SGKoihTRtqVgjHmdeA/gEeBh4GNQHTIPgYww48GY8ztxpg1xpg1dXV1qRY37cys8NLVH6YvFOGpnS24CwZMzYW1w0f9KYqiTCUZyT4yxvzcGHOmMeYioAPYBTSJSAOA/bt5tNfIVRrshnY7j/vZcKiDK1bOjG9bUFucKbEURckTMqIURKTe/j0PK57wO+BB4IP2Lh8EHsiEbJlmZrlVh3DP+sPEDLzvrHnxbcUeLUBXFCW1ZOoqc6+I1ABh4GPGmE4R+TZwj4jcCBwErsuQbBnFsRTuevkwDRVezltcM8YRiqIoU0dGlIIx5sIka23A8AT+PCNxHsK1p8+mwCV88k1LmVetriNFUVKP+iOyDK+7gN/ceA4PbzvG352/AIBPvumkzAqlKEreoEohC7lgaS0XLK3NtBiKouQh2vtIURRFiaNKQVEURYmjSkFRFEWJo0pBURRFiaNKQVEURYmjSkFRFEWJo0pBURRFiaNKQVEURYkjVpfq6YmItGD1SZostUDrFImTKqaDjKByTiXTQUZQOaeSdMs43xiTdPbAtFYKJ4qIrDfGrMm0HKMxHWQElXMqmQ4ygso5lWSTjOo+UhRFUeKoUlAURVHi5LtSuD3TAoyD6SAjqJxTyXSQEVTOqSRrZMzrmIKiKIoymHy3FBRFUZQEVCkoiqIocfJSKYjIlSKyU0T2iMgXMi1PIiJyQES2iMhGEVlvr1WLyFoR2W3/rsqAXHeISLOIbE1YSyqXWPyPfX43i8gZGZTx6yJy1D6fG0XkqoRtN9sy7hSRN6dDRvt954rIkyKyXUS2icgn7PWsOZ+jyJhV51NEvCLysohssuX8hr2+UEResuX5vYh47PUi+/kee/uCDMt5p4jsTzifq+31jHyHADDG5NUPUADsBRYBHmATsCLTciXIdwCoHbL2n8AX7MdfAP4jA3JdBJwBbB1LLuAq4K+AAOcCL2VQxq8Dn02y7wr7f18ELLQ/EwVpkrMBOMN+XAbssuXJmvM5ioxZdT7tc1JqP3YDL9nn6B7genv9x8BH7cf/BPzYfnw98Ps0/c9HkvNO4N1J9s/Id8gYk5eWwtnAHmPMPmNMCLgbuCbDMo3FNcAv7ce/BN6RbgGMMeuA9iHLI8l1DfArY/EiUCkiDRmScSSuAe42xgSNMfuBPVifjZRjjDlmjNlgP/YDrwOzyaLzOYqMI5GR82mfkx77qdv+McClwB/t9aHn0jnHfwQuExHJoJwjkZHvEOSn+2g2cDjh+RFG/7CnGwM8KiKvishN9toMY8wx+/FxYEZmRBvGSHJl2zn+Z9sEvyPB9ZYVMtrui9Ox7hyz8nwOkRGy7HyKSIGIbASagbVYVkqnMSaSRJa4nPb2LqAmE3IaY5zzeYt9Pv9bRIqGymmTtvOZj0oh27nAGHMG8BbgYyJyUeJGY9mWWZdHnK1yAbcBi4HVwDHgu5kVZwARKQXuBT5pjOlO3JYt5zOJjFl3Po0xUWPMamAOlnWyPMMiJWWonCKyCrgZS96zgGrg8xkUEchPpXAUmJvwfI69lhUYY47av5uB+7E+5E2O6Wj/bs6chIMYSa6sOcfGmCb7yxgDfsqASyOjMoqIG+ti+1tjzH32cladz2QyZuv5tGXrBJ4EzsNytxQmkSUup729AmjLkJxX2m46Y4wJAr8gC85nPiqFV4CldnaCByvY9GCGZQJAREpEpMx5DFwBbMWS74P2bh8EHsiMhMMYSa4HgRvsDIpzga4Et0haGeKHvRbrfIIl4/V2NspCYCnwcppkEuDnwOvGmO8lbMqa8zmSjNl2PkWkTkQq7cc+4HKs+MeTwLvt3YaeS+ccvxt4wrbKMiHnjoSbAMGKeySez8x8h9IV0c6mH6zI/i4s3+OXMi1PglyLsDI4NgHbHNmwfJ6PA7uBx4DqDMh2F5a7IIzl37xxJLmwMiZ+aJ/fLcCaDMr4a1uGzVhftIaE/b9ky7gTeEsaz+UFWK6hzcBG++eqbDqfo8iYVecTOBV4zZZnK/BVe30RllLaA/wBKLLXvfbzPfb2RRmW8wn7fG4FfsNAhlJGvkPGGG1zoSiKogyQj+4jRVEUZQRUKSiKoihxVCkoiqIocVQpKIqiKHFUKSiKoihxVCkoygQQkajdzXKb3fHyMyIy6vdIRBaIyN+kS0ZFORFUKSjKxOg3xqw2xqzEKkB6C/C1MY5ZAKhSUKYFWqegKBNARHqMMaUJzxdhVcnXAvOxirtK7M3/bIx5XkReBE4G9mN16Pwf4NvAxVitpn9ojPlJ2v4IRRkFVQqKMgGGKgV7rRNYBviBmDEmICJLgbuMMWtE5GKsGQRvtfe/Cag3xnzL7or5HPAeY7WcVpSMUjj2LoqijBM3cKs9PSsKnDTCflcAp4qI05unAqtXkCoFJeOoUlCUE8B2H0WxOpp+DWgCTsOK1wVGOgz4uDHmkbQIqSgTQAPNijJJRKQOa9Tjrcbyw1YAx4zVVvoDWKNfwXIrlSUc+gjwUbs1NSJykt0VV1EyjloKijIxfPb0LDcQwQosO62lfwTcKyI3AA8Dvfb6ZiAqIpuwZvL+ACsjaYPdMrmFDIxYVZRkaKBZURRFiaPuI0VRFCWOKgVFURQljioFRVEUJY4qBUVRFCWOKgVFURQljioFRVEUJY4qBUVRFCXO/wdzzBymqFxIWQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yqkjpADQ_btB",
        "colab_type": "text"
      },
      "source": [
        "### Section Three -- Reinforcement Learning\n",
        "\n",
        "This section follows closely the [TF-Agent documentation](https://www.tensorflow.org/agents)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EiamAtrtzi3Y",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "4f895083-c3ba-4a01-bcee-0e2978f4956a"
      },
      "source": [
        "!pip install -q tf-agents"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\u001b[K     |████████████████████████████████| 1.1MB 2.8MB/s \n",
            "\u001b[?25h"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z7glhOFbzxaW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "from tf_agents.environments import  gym_wrapper           # wrap OpenAI gym\n",
        "from tf_agents.environments import tf_py_environment      # gym to tf gym\n",
        "from tf_agents.networks import q_network                  # Q net\n",
        "from tf_agents.agents.dqn import dqn_agent                # DQN Agent\n",
        "from tf_agents.replay_buffers import tf_uniform_replay_buffer      # replay buffer\n",
        "from tf_agents.trajectories import trajectory              # s->s' trajectory\n",
        "from tf_agents.utils import common                       # loss function"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mfGnbc0eAvYu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Hyper-parameters\n",
        "num_iterations = 20000 # @param {type:\"integer\"}\n",
        "\n",
        "collect_steps_per_iteration = 10  # @param {type:\"integer\"}\n",
        "replay_buffer_max_length = 100000  # @param {type:\"integer\"}\n",
        "batch_size = 256  # @param {type:\"integer\"}\n",
        "\n",
        "learning_rate = 1e-3  # @param {type:\"number\"}\n",
        "num_eval_episodes = 10  # @param {type:\"integer\"}\n",
        "\n",
        "eval_interval = 1000  # @param {type:\"integer\"}\n",
        "log_interval = 200  # @param {type:\"integer\"}"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FtV-1MkFh3Q",
        "colab_type": "text"
      },
      "source": [
        "#### Section 3.1 Define Environment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cWkMWaUjz915",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "04819377-3107-4a59-be32-7bbf80b4aa96"
      },
      "source": [
        "train_env_gym = AmeriOptionEnv()\n",
        "eval_env_gym = AmeriOptionEnv()\n",
        "\n",
        "train_env_wrap = gym_wrapper.GymWrapper(train_env_gym)\n",
        "eval_env_wrap = gym_wrapper.GymWrapper(eval_env_gym)\n",
        "\n",
        "train_env  = tf_py_environment.TFPyEnvironment(train_env_wrap)\n",
        "eval_env = tf_py_environment.TFPyEnvironment(eval_env_wrap)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n",
            "  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RzH7SYbBFrRf",
        "colab_type": "text"
      },
      "source": [
        "#### Section 3.2 Define Q Net"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J7jpKho5w7kM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "fc_layer_params = (100,)\n",
        "\n",
        "q_net = q_network.QNetwork(\n",
        "    train_env.observation_spec(),\n",
        "    train_env.action_spec(),\n",
        "    fc_layer_params=fc_layer_params)"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N2Y84kHxFu8a",
        "colab_type": "text"
      },
      "source": [
        "#### Section 3.3 Define DQN Agent"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ed9QJG1QxFZS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate=learning_rate)\n",
        "\n",
        "train_step_counter = tf.Variable(0)\n",
        "\n",
        "agent = dqn_agent.DqnAgent(\n",
        "    train_env.time_step_spec(),\n",
        "    train_env.action_spec(),\n",
        "    q_network=q_net,\n",
        "    optimizer=optimizer,\n",
        "    td_errors_loss_fn=common.element_wise_squared_loss,\n",
        "    train_step_counter=train_step_counter)\n",
        "\n",
        "agent.initialize()"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3fzyCueeF2lW",
        "colab_type": "text"
      },
      "source": [
        "#### Section 3.4 - Construct Experience Replay Buffer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0jcMGgYuyTv8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "replay_buffer = tf_uniform_replay_buffer.TFUniformReplayBuffer(\n",
        "    data_spec=agent.collect_data_spec,\n",
        "    batch_size=train_env.batch_size,\n",
        "    max_length=replay_buffer_max_length)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JFYE2xC8ydt0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Data Collection\n",
        "\n",
        "def collect_step(environment, policy, buffer):\n",
        "  time_step = environment.current_time_step()\n",
        "  action_step = policy.action(time_step)\n",
        "  next_time_step = environment.step(action_step.action)\n",
        "  traj = trajectory.from_transition(time_step, action_step, next_time_step)\n",
        "\n",
        "  # Add trajectory to the replay buffer\n",
        "  buffer.add_batch(traj)\n",
        "\n",
        "def collect_data(env, policy, buffer, steps):\n",
        "  for _ in range(steps):\n",
        "    collect_step(env, policy, buffer)"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EFfjRLuX0gtX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "f7c3a7a7-fa70-4dcf-a1e1-45e2927f3d31"
      },
      "source": [
        "# Fetch experience\n",
        "\n",
        "dataset = replay_buffer.as_dataset(\n",
        "    num_parallel_calls=3, \n",
        "    sample_batch_size=batch_size, \n",
        "    num_steps=2).prefetch(3)\n",
        "\n",
        "iterator = iter(dataset)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/operators/control_flow.py:1004: ReplayBuffer.get_next (from tf_agents.replay_buffers.replay_buffer) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use `as_dataset(..., single_deterministic_pass=False) instead.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "On5yCJ0mF-mv",
        "colab_type": "text"
      },
      "source": [
        "#### Section 3.5 - Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZF9NgvO6GBUP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# eval_env evaluation\n",
        "def compute_avg_return(environment, policy, num_episodes=10):\n",
        "\n",
        "  total_return = 0.0\n",
        "  for _ in range(num_episodes):\n",
        "\n",
        "    time_step = environment.reset()\n",
        "    episode_return = 0.0\n",
        "\n",
        "    while not time_step.is_last():\n",
        "      action_step = policy.action(time_step)\n",
        "      time_step = environment.step(action_step.action)\n",
        "      episode_return += time_step.reward\n",
        "    total_return += episode_return\n",
        "\n",
        "  avg_return = total_return / num_episodes\n",
        "  return avg_return.numpy()[0]"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H3aEpL9bzOn9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "391d080d-50eb-43f5-95f3-5a01773749c2"
      },
      "source": [
        "# step 5 - training - takes a while\n",
        "# (Optional) Optimize by wrapping some of the code in a graph using TF function.\n",
        "agent.train = common.function(agent.train)\n",
        "\n",
        "# Reset the train step\n",
        "agent.train_step_counter.assign(0)\n",
        "\n",
        "# Evaluate the agent's policy once before training.\n",
        "avg_return = compute_avg_return(eval_env, agent.policy, num_eval_episodes)\n",
        "returns = [avg_return]\n",
        "\n",
        "for _ in range(num_iterations):\n",
        "\n",
        "  # Collect a few steps using collect_policy and save to the replay buffer.\n",
        "  for _ in range(collect_steps_per_iteration):\n",
        "    collect_step(train_env, agent.collect_policy, replay_buffer)\n",
        "\n",
        "  # Sample a batch of data from the buffer and update the agent's network.\n",
        "  experience, unused_info = next(iterator)\n",
        "  train_loss = agent.train(experience).loss\n",
        "\n",
        "  step = agent.train_step_counter.numpy()\n",
        "\n",
        "  if step % log_interval == 0:\n",
        "    print('step = {0}: loss = {1}'.format(step, train_loss))\n",
        "\n",
        "  if step % eval_interval == 0:\n",
        "    avg_return = compute_avg_return(eval_env, agent.policy, num_eval_episodes)\n",
        "    print('step = {0}: Average Return = {1}'.format(step, avg_return))\n",
        "    returns.append(avg_return)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py:201: calling foldr_v2 (from tensorflow.python.ops.functional_ops) with back_prop=False is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "back_prop=False is deprecated. Consider using tf.stop_gradient instead.\n",
            "Instead of:\n",
            "results = tf.foldr(fn, elems, back_prop=False)\n",
            "Use:\n",
            "results = tf.nest.map_structure(tf.stop_gradient, tf.foldr(fn, elems))\n",
            "step = 200: loss = 59.832584381103516\n",
            "step = 400: loss = 238.35992431640625\n",
            "step = 600: loss = 626.5194091796875\n",
            "step = 800: loss = 1394.18701171875\n",
            "step = 1000: loss = 1666.5810546875\n",
            "step = 1000: Average Return = 7.270626068115234\n",
            "step = 1200: loss = 1957.728759765625\n",
            "step = 1400: loss = 1191.81884765625\n",
            "step = 1600: loss = 1217.3583984375\n",
            "step = 1800: loss = 1168.137451171875\n",
            "step = 2000: loss = 1351.412109375\n",
            "step = 2000: Average Return = 8.32473087310791\n",
            "step = 2200: loss = 1633.104248046875\n",
            "step = 2400: loss = 1531.072021484375\n",
            "step = 2600: loss = 1716.333251953125\n",
            "step = 2800: loss = 2605.77685546875\n",
            "step = 3000: loss = 3474.59033203125\n",
            "step = 3000: Average Return = 8.851873397827148\n",
            "step = 3200: loss = 3067.1572265625\n",
            "step = 3400: loss = 2595.955810546875\n",
            "step = 3600: loss = 2869.69287109375\n",
            "step = 3800: loss = 3171.66796875\n",
            "step = 4000: loss = 2977.46728515625\n",
            "step = 4000: Average Return = 7.773991584777832\n",
            "step = 4200: loss = 2923.05224609375\n",
            "step = 4400: loss = 3331.69677734375\n",
            "step = 4600: loss = 3646.66845703125\n",
            "step = 4800: loss = 4468.64306640625\n",
            "step = 5000: loss = 3540.42236328125\n",
            "step = 5000: Average Return = 6.310206413269043\n",
            "step = 5200: loss = 3945.1953125\n",
            "step = 5400: loss = 3682.572998046875\n",
            "step = 5600: loss = 2957.159912109375\n",
            "step = 5800: loss = 3450.229248046875\n",
            "step = 6000: loss = 3282.22509765625\n",
            "step = 6000: Average Return = 3.704317092895508\n",
            "step = 6200: loss = 4112.20166015625\n",
            "step = 6400: loss = 3733.779296875\n",
            "step = 6600: loss = 3758.38525390625\n",
            "step = 6800: loss = 3866.689453125\n",
            "step = 7000: loss = 4750.37255859375\n",
            "step = 7000: Average Return = 4.393885135650635\n",
            "step = 7200: loss = 4063.758544921875\n",
            "step = 7400: loss = 4408.9833984375\n",
            "step = 7600: loss = 3571.4580078125\n",
            "step = 7800: loss = 4285.1865234375\n",
            "step = 8000: loss = 4719.5146484375\n",
            "step = 8000: Average Return = 9.878602981567383\n",
            "step = 8200: loss = 3374.521728515625\n",
            "step = 8400: loss = 3679.5966796875\n",
            "step = 8600: loss = 3294.890380859375\n",
            "step = 8800: loss = 4248.90234375\n",
            "step = 9000: loss = 4398.8662109375\n",
            "step = 9000: Average Return = 10.974069595336914\n",
            "step = 9200: loss = 4877.45654296875\n",
            "step = 9400: loss = 4924.79296875\n",
            "step = 9600: loss = 6406.14404296875\n",
            "step = 9800: loss = 6251.0146484375\n",
            "step = 10000: loss = 5566.1396484375\n",
            "step = 10000: Average Return = 7.104763984680176\n",
            "step = 10200: loss = 5940.2431640625\n",
            "step = 10400: loss = 6727.74951171875\n",
            "step = 10600: loss = 5354.83056640625\n",
            "step = 10800: loss = 6944.0458984375\n",
            "step = 11000: loss = 6237.296875\n",
            "step = 11000: Average Return = 9.988594055175781\n",
            "step = 11200: loss = 7092.5\n",
            "step = 11400: loss = 6782.16259765625\n",
            "step = 11600: loss = 7195.4892578125\n",
            "step = 11800: loss = 6972.521484375\n",
            "step = 12000: loss = 7839.734375\n",
            "step = 12000: Average Return = 6.589081764221191\n",
            "step = 12200: loss = 7124.73095703125\n",
            "step = 12400: loss = 9350.556640625\n",
            "step = 12600: loss = 8233.65625\n",
            "step = 12800: loss = 8683.29296875\n",
            "step = 13000: loss = 8907.5009765625\n",
            "step = 13000: Average Return = 5.671632289886475\n",
            "step = 13200: loss = 9058.08203125\n",
            "step = 13400: loss = 9229.42578125\n",
            "step = 13600: loss = 9006.544921875\n",
            "step = 13800: loss = 8510.9853515625\n",
            "step = 14000: loss = 9647.751953125\n",
            "step = 14000: Average Return = 5.804669380187988\n",
            "step = 14200: loss = 12039.1845703125\n",
            "step = 14400: loss = 10787.4482421875\n",
            "step = 14600: loss = 9600.1943359375\n",
            "step = 14800: loss = 12051.8154296875\n",
            "step = 15000: loss = 12085.916015625\n",
            "step = 15000: Average Return = 1.3238499164581299\n",
            "step = 15200: loss = 11897.4033203125\n",
            "step = 15400: loss = 12195.90234375\n",
            "step = 15600: loss = 13908.0703125\n",
            "step = 15800: loss = 13883.7294921875\n",
            "step = 16000: loss = 16726.701171875\n",
            "step = 16000: Average Return = 4.567616939544678\n",
            "step = 16200: loss = 18005.771484375\n",
            "step = 16400: loss = 16287.5205078125\n",
            "step = 16600: loss = 14791.625\n",
            "step = 16800: loss = 17904.505859375\n",
            "step = 17000: loss = 15225.501953125\n",
            "step = 17000: Average Return = 13.814481735229492\n",
            "step = 17200: loss = 19509.3359375\n",
            "step = 17400: loss = 14184.224609375\n",
            "step = 17600: loss = 20179.828125\n",
            "step = 17800: loss = 20171.0546875\n",
            "step = 18000: loss = 20839.328125\n",
            "step = 18000: Average Return = 5.019478797912598\n",
            "step = 18200: loss = 18506.23828125\n",
            "step = 18400: loss = 26637.939453125\n",
            "step = 18600: loss = 20928.078125\n",
            "step = 18800: loss = 20780.12109375\n",
            "step = 19000: loss = 18977.041015625\n",
            "step = 19000: Average Return = 3.841068983078003\n",
            "step = 19200: loss = 18923.05078125\n",
            "step = 19400: loss = 23536.375\n",
            "step = 19600: loss = 18502.41796875\n",
            "step = 19800: loss = 16992.486328125\n",
            "step = 20000: loss = 20572.65625\n",
            "step = 20000: Average Return = 4.349373817443848\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l5sgVeOCz1FH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "outputId": "ef8a60c6-4315-4791-92fb-dc373be37538"
      },
      "source": [
        "# plot training iterations\n",
        "iterations = range(0, num_iterations + 1, eval_interval)\n",
        "plt.plot(iterations, returns)\n",
        "plt.ylabel('Average Return')\n",
        "plt.xlabel('Iterations')\n",
        "plt.ylim(top=20)"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.6993183255195617, 20.0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEKCAYAAADn+anLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hU95X4//dRBxUQqIBAoEIRxQbbAoPBNja4ro0dJ7GNE5fYCW5xijfF2WzKN+W32WyKd+3E3XGJe7fjigkYFxDNNNNVQIBADRBIqJ/fH3NFBqGRRtI0ac7reebR6M6de49G0py5n3I+oqoYY4wxHYkIdgDGGGNClyUJY4wxHlmSMMYY45ElCWOMMR5ZkjDGGOORJQljjDEe+S1JiEimiCwRkc0i8oWIfNfZPkREFonIDudrsofn3+jss0NEbvRXnMYYYzwTf82TEJHhwHBVXSsiicAa4ErgJqBaVX8nIvcAyar643bPHQKsBvIBdZ57hqoe9EuwxhhjOuS3KwlVLVPVtc79I8AWYARwBfCks9uTuBJHexcBi1S12kkMi4CL/RWrMcaYjkUF4iQikgWcBhQA6apa5jy0H0jv4CkjgFK37/c42zo69kJgIUB8fPwZeXl5vgnaGGPCwJo1aypVNdXT435PEiKSALwCfE9Va0Tk+GOqqiLSq/YuVX0YeBggPz9fV69e3ZvDGWNMWBGRXZ097tfRTSISjStBPKOqrzqbDzj9FW39FuUdPHUvkOn2/UhnmzHGmADy5+gmAR4Dtqjqn9weehNoG610I/BGB09/H7hQRJKd0U8XOtuMMcYEkD+vJGYB1wPni8g653Yp8DvgAhHZAcxzvkdE8kXkUQBVrQZ+Daxybr9ythljjAkgvw2BDQbrkzDGmO4RkTWqmu/pcZtxbYwxxiNLEsYYYzyyJGGMMcYjSxLGGGM8siRhjDHGI0sSxhhjPLIkYYwxxiNLEsYYYzyyJGGMMcYjSxLGGGM8siRhjDHGI0sSxhhjPLIkYYwxxiNLEsYYYzyyJGGMMcYjSxLGGGM8siRhjDHGI0sSxhhjPLIkYYwxxqMofx1YRB4HLgPKVXWys+0FYLyzy2DgkKpO7eC5JcARoAVo7mz9VWOMMf7jtyQBPAHcDzzVtkFVr2m7LyJ/BA538vzzVLXSb9EZY4zpkt+ShKouE5Gsjh4TEQGuBs731/mNMcb0XrD6JM4GDqjqDg+PK/CBiKwRkYUBjMsYY4wbfzY3dWYB8Fwnj89W1b0ikgYsEpGtqrqsox2dJLIQYNSoUb6P1BhjwljAryREJAq4CnjB0z6qutf5Wg68BkzvZN+HVTVfVfNTU1N9Ha4xxoS1YDQ3zQO2quqejh4UkXgRSWy7D1wIbApgfMYYYxx+SxIi8hywHBgvIntE5BbnoWtp19QkIhki8o7zbTrwiYisB1YCb6vqe/6K0xhjjGf+HN20wMP2mzrYtg+41LlfBEzxV1zGGGO8ZzOujTHGeGRJwhhjjEeWJIwxxnhkScIYY4xHliSMMcZ4ZEnCGGOMR5YkjDHGeGRJwhhjjEeWJIwxxnhkScIYY4xHliSMMcZ4ZEnCGGOMR5YkjDHGeGRJwhhjjEeWJIwxxnhkScIYY4xHliSMMcZ4ZEnCGGOMR5YkjDHGeGRJwhhjjEd+SxIi8riIlIvIJrdtvxSRvSKyzrld6uG5F4vINhHZKSL3+CtGY4wxnfPnlcQTwMUdbP+zqk51bu+0f1BEIoG/AJcAE4EFIjLRj3EaY4zxwG9JQlWXAdU9eOp0YKeqFqlqI/A8cIVPgzPGGOOVYPRJfFtENjjNUckdPD4CKHX7fo+zrUMislBEVovI6oqKCl/HaowxYS3QSeIBIBeYCpQBf+ztAVX1YVXNV9X81NTU3h7OGGOMm4AmCVU9oKotqtoKPIKraam9vUCm2/cjnW3GGGMCLKBJQkSGu337JWBTB7utAsaKSLaIxADXAm8GIj5jjDEnivLXgUXkOWAOkCIie4BfAHNEZCqgQAlwq7NvBvCoql6qqs0i8m3gfSASeFxVv/BXnMYYYzwTVQ12DD6Tn5+vq1evDnYYxhjTZ4jIGlXN9/S4zbg2xhjjkSUJY4wxHlmSMMYY45ElCWOMMR5ZkjDGGOORJQljjDEeWZIwxhigpr6Jn762kZr6pmCHElIsSRhjDPDpjkqeKdjNJzsqgx1KSLEkYYwxQHFVLQBFFUeDHElosSRhjDFAcYUrSRQ6X42LV7WbROQsIMt9f1V9yk8xGWNMwJXYlUSHukwSIvI0rjUg1gEtzmYFLEkYY/qN4so6AIoqalFVRCTIEYUGb64k8oGJ2p8qARpjjJsj9U1UHm0gY1Ac+w7XU3GkgbSkuGCHFRK86ZPYBAzzdyDGGBMsJc5VxNwJ6YD1S7jzJkmkAJtF5H0RebPt5u/AjDEmUIoqXf0Q509IO+F7411z0y/9HYQxxgRTSWUdIjAjeygDoiMpLLcriTadJgkRiQQeUtW8AMVjjDEBV1JVS8agAQyIiSQ7Jd6uJNx02tykqi3ANhEZFaB4jDEm4Ioqa8lOiQcgJzWeIuuTOM6bPolk4AsRWWx9EsaY/kZVKa44SlbKQAByUxMoPVhHfVNLF88MD970SfysJwcWkceBy4ByVZ3sbPsf4HKgESgEvqGqhzp4bglwBNe8jObO1l81xpjeOFjXRE19M9kpCYDrSkIVdlXVMX5YYpCjC74uryRU9aOObl4c+wng4nbbFgGTVfVUYDvwk06ef56qTrUEYYzxp+JKV9NSttuVBNjM6zZdJgkROSIiNc6tXkRaRKSmq+ep6jKgut22D1S12fl2BTCyR1EbY4yPlDhJImuoq0+irW+i0JIE4EVzk6oev94S1zz1K4AZPjj3zcALnk4LfCAiimt01cOeDiIiC4GFAKNGWf+6MaZ7iitriYwQMoe4riTiY6MYPijOOq8d3aoCqy6vAxf15qQi8lOgGXjGwy6zVfV04BLgThE5p5OYHlbVfFXNT01N7U1YxpgwVFxVS2byAKIj//V2mJMaT2GlJQnwrsDfVW7fRuCq5VTf0xOKyE24OrTneqoHpap7na/lIvIaMB1Y1tNzGmOMJ8UV/xr+2iY3NYHX1u61Qn94N7rpcrf7zUAJrianbhORi4EfAeeqap2HfeKBCFU94ty/EPhVT85njDGdUVVKqmo5M2fICdtzUuI50tBMxdEG0hLDu9CfN0niUVX91H2DiMwCyjt7kog8B8wBUkRkD/ALXKOZYoFFTnZeoaq3iUiGc55LgXTgNefxKOBZVX2vWz+VMcZ4oeJIA3WNLSddSeQcH+FUa0nCi33uA073YtsJVHVBB5sf87DvPuBS534RMMWLuIwxpleKjg9/bdfclOZKEoUVR5mRMzTgcYUSj0lCRGYCZwGpInK320NJQKS/AzPGGH9rP/y1zfCkOOKiI2yEE51fScQACc4+7tMOa4Cv+DMoY4wJhOLKWmKiIsgYPOCE7RERQnZKgk2oo5Mk4cyq/khEnlDVXSIy0FNnszHG9EXFlbWMHjKQyIiTRzDlpsazYc/hIEQVWryZJ5EhIpuBrQAiMkVE/urfsIwxxv+KK2vJatcf0SYnNYE9B+toaA7vQn/eJIl7cU2eqwJQ1fWAx8ltxhjTF7S2Kruq68jxkCRyU+NpdQr9hTOvZlyramm7TeGdWo0xfd6+w8dobG71eCXRVuivsDy8+yW8GQJbKiJnASoi0cB3gS3+DcsYY/yr2MPw1zZt24vCvDyHN1cStwF3AiOAvcBU4A5/BmWMMf5W0kWSiI+NYlhSXNhXg/WmCmwl8LW270UkGVeS+K0f4zLGGL8qqqxlYEwkaYmxHvfJSY2nMMznSni8khCRTBF5WET+ISK3iEi8iPwB2AakBS5EY4zxvZLKWrKGxndawC831TVXwkMt0rDQWXPTU8A+XCU4JgOrcTU5naqq3w1AbMaEhJr6Jn79j82sLz1ppV3Th5VU1XlsamqTkxrPkfpmKo82Biiq0NNZkhiiqr9U1fdV9fu4Zl1/TVX3Byg2Y4KusOIoV/7lUx77pJg/f7g92OEYH2lqaWV3dR1ZzpKlnrQV+gvnfolOO65FJFlEhojIEFzzJAa5fW9Mv7ZkazlX3v8ph+uaOD8vjU92VFJdG76fKPuTPQeP0dKqZKckdLpfbqozwimM+yU667geBKwB3Bvs1jpfFcjxV1DGBJOq8telhfzhg21MHJ7Ewzfkc7iuiX9uLeedjWV8fcboYIdoeqm40nVlkN3FlUTGoAFOob/wvZLorHZTVgDjMCYk1DU286OXN/CPDWVcPiWD33/5VAbERJIxKI6xaQm8uW6fJYl+oLjSNYu6qyuJiAgha2i8NTcZY2DPwTq+8sBy3t5Yxj2X5PF/105lQIyrKr6IMH9KBitLqtl36FiQIzW9VVJZS1JcFMkDo7vcNzctIawn1FmSMAZYUVTF/Ps/pfRgHY/fOI3bzs09aWjk5VMyAHhr/b5ghGh8qLjSta61N+tX56bEU1odvoX+LEmYsKaqPLW8hK8/WkDywGjeuHMW5+V1PA0oKyWeKZmDedOSRJ/XliS8kZOaENaF/rxKEiIyW0S+4dxPFZFs/4ZljP81NLfwk1c38vM3vuCccam8dues40MePZk/JYMv9tWwM8yLvvVl9U0t7Dt8zGNhv/Zyj693HZ6/8y6ThIj8Avgx8BNnUzTwd28OLiKPi0i5iGxy2zZERBaJyA7na7KH597o7LNDRG705nzGeKv8SD3XPVLA86tKufO8XB65IZ+kuK7bpy87dTgi2NVEH7a7ug5VzzWb2st2hsGGa3kOb64kvgTMB2oBVHUfJy5n2pkngIvbbbsHWKyqY4HFzvcncOZh/AI4E5gO/MJTMjGmu9aXHmL+fZ+yeV8Nf7nudH54UV6HK5N1JD0pjpk5Q3lr/b6wLtXQl3VV/bW9hNgo0pNiw3aEkzdJolFd/w0KICLevbKAqi4DqtttvgJ40rn/JHBlB0+9CFikqtWqehBYxMnJxphue3XtHr760HIiI4RXbj+Lfzt1eLePMX9KBsWVtWzaW+OHCE+0cc9hnl6xy+/nCSdtScLb5iZoq+FkVxKevCgiDwGDReRbwIfAI704Z7qqljn39wPpHewzAnBf6GiPs+0kIrJQRFaLyOqKiopehGX6M1Xlt29v5u4X13P6qMG8+e1ZTMxI6tGxLpk8nOhI4Y11e30c5YlUlZ+8toGfvb7p+Bub6b2SylpSEmK8al5sk5MaH7aF/rpMEqr6B+Bl4BVgPPBzVb3PFyd3v0LpxTEeVtV8Vc1PTU31RVimH1peWMUjHxdz3ZmjePqWMxma4Lk8dFcGDYzm3HFpvLVhHy2t/nvT+GRn5fGrledX7fbbecJNkVP9tTtyUhKoqW+mKgzLsni7fOkiVf2hqv5AVRf18pwHRGQ4gPO1vIN99gKZbt+PdLYZ0yPLi6qIEPjJJXlER/Z+5Pf8qRkcqGlgZXH71lTfeWBpIelJsZw3PpWXV++hsbnVb+cKJyXdGP7aJjctfJcy9WZ00xERqWl3KxWR10SkJ/Wb3gTaRivdCLzRwT7vAxc6BQaTgQudbcb0yIqiKiaPGERiN5oYOjNvQhoDYyL9NsppXekhPius4puzc7jhrCyqahtZtPmAX84VTmobmik/0tCt/giAnDBeytSbj1T3Aj/E1ScwEvgB8CzwPPB4Z08UkeeA5cB4EdkjIrcAvwMuEJEdwDzne0QkX0QeBVDVauDXwCrn9itnmzHdVt/UwvrSw8zIGeqzYw6MieKCiem8u6nML5/wH1xaSFJcFAvOHMU5Y1MZMXiANTn5QHdHNrUZMXgAsVHhWejPmyQxX1UfUtUjqlqjqg8DF6nqC0Cnw1JVdYGqDlfVaFUdqaqPqWqVqs5V1bGqOq/tzV9VV6vqN92e+7iqjnFuf+vVT2nC2trdB2lsaeXMbN9WuJ8/JYNDdU18vMO3AyZ2lh/l/c37ufGsLBJio4iMEK6ZlsnHOyrZHaazfn2lpKpnSSIiQshOCc+lTL1JEnUicrWIRDi3q4F657Hw6+o3fU5BUTUikJ/l2yRx9thUBg+M9nmT08PLComNiuCms7KOb7s6P5MIsQ7s3ip23uS723EN/1rKNNx4kyS+BlyPq4P5gHP/6yIyAPi2H2MzxicKiquYODyJQQN80x/RJiYqgksmD2fR5gPUNTb75Jj7D9fz2ud7uSY/84QRWMMGxXF+Xhovrt5DU4t1YPdUcVUtwwfFHa/u2x05qfGUHjwWdoX+vBkCW6Sql6tqiqqmOvd3quoxVf0kEEEa01MNzS18vvsQZ2b7rj/C3fwpGdQ1tvDhlo4G6XXfY58U0arwzbNPHhOyYPooKo82sNhH5wpHJT0Y/tomNzWBllYNuyY/b0Y3xYnInSLyV6cW0+Mi0mmHtTGhYn3pYRqaWzkzxz8r7k7PHsKwpDjeXNf7JqdDdY08W7Cb+VMyyBxy8opp545LZfigOJ5baU1OPVVcWdvtkU1tcsK0hpM3zU1PA8Nwlcr4CNcIpyP+DMoYXykoqgJguo/7I9pERgiXnTqcj7aXc6iudxOtnl6+i9rGFm49t+OR5VGREXw1P5NlOyoorQ6vT7O+cKiukYN1TceHs3ZX9vFhsOHVL+FNkhijqj8DalX1SeDfcBXeMybkFRRXkzcskeT4GL+dY/7UDJpalPc27e/xMY41tvC3z0qYm5dG3jDP5UKumeaaY/ri6lKP+5iO9aRmk7vEuGhXob9yu5Jor8n5ekhEJgODgI5XZTEBUXb4GPct3sG3nlrNhj2Hgh1OyGpqaWXNroM+H/ra3ikjBpGdEt+rUU4vri6luraR2+fkdrrfiMEDmDMulRdXl9JsHdjd8q/hryc35XkrJyXBriQ68LAz6/k/cc2W3gz8t1+jMidpamnl/S/2c/MTq5j1u3/yx0XbWVFUxVceXM4ra/YEO7yQtGHPYY41tXCmDyfRdUREuHxKBsuLqjhQU9/1E9ppamnl4WVF5I9O9mqY7oLpozhQ08CSbVbQsjuKK+uIEDrs7/GWq9BfbVgV+us0SYhIBFCjqgdVdZmq5qhqmqo+FKD4wl5xZS2/e3crM//rn9z69Bq+2HeYO+aMYdkPz2PpD+aQPzqZf39pPb988wsbGtlOQbHTH+HnKwlwjXJShX9sKOt653b+sWEfew8d6/Iqos35eWmkJcZaB3Y3FVfWMiJ5ALFR3R/+2iY3NYHDx5rCqtBfVGcPqmqriPwIeDFA8Rhc7dPvbirj+VWlrCyuJjJCOD8vjWunZXLuuFSi3ArUPXXzdP7r3a089kkxm8tci+ikJva8wml/UlBUzZi0BFJ6UfHVW2PSEpiUkcSb6/Zyy2zvV/dtbVUeWFrI+PREzhvvXStuVGQEV+dn8telO9l36BgZgwf0NOyw4irs1/nytF1pG+FUVFEbkL+rUOBNc9OHIvIDEcl0lh4d4qwcZ3xs097D/Oz1TUz//z7k7hfXU15Tz48vzmP5PefzyA35zJ2QfkKCANcbxs8um8i910xlw55DzL//E9aXWj9Fc0srq0uq/d4f4W7+lAzW7zlMSTeKwC3ZVs72A0e5bU4OEV6ujgeuDmzFOrC9paoUV9aSPbTnTU0Qnutdd3ol4bjG+Xqn2zYFelIB1rRTU9/EG+v28cKq3WzaW0NsVASXnjKca6Zlcmb2EES8e+O48rQRjElL4Nan1/DVh5bzmysnc3V+ZtdP7Ke+2FdDbaP/+yPcXT4lg/96dytvrd/HXXPHevWcB5YWMmLwAC47NaNb58ocMpCzx6bywqpS7jp/rNfLr4aryqONHG1o7vHIpjYZTqG/cFrK1JsZ19kd3CxB9FJrq3Lf4h1M/+2H/Oz1TbS0wq+umMTK/5jHn6+ZyoycoV4niDaTRwzirbtmMy0rmR+97FrRLFzXIGjrj5gRwCuJjMEDmJ41hDe8XP96VUk1q3cdZOE5OT1a42LBtEzKDtfz0Xabgd2Vnhb2ay/SKfQXTkuZejPjeqCI/KeIPOx8P1ZELvN/aP3X4bombnlyFX9ctJ25eem89e3ZvPOd2dwwM4tBA3tXX2hIfAxPfmM6t56Tw9MrdvG1R1dQfqT7I276uoKiarJT4klLigvoeS+fmsHO8qNsKet6vumDSwsZEh/T4yu+eRPTSUmI5bmV1uTUlbbCfr1NEuCMcAqjdSW8+fjyN6AROMv5fi/wG79F1M9t2nuYy+7/mE92VvLrKydz/3WnccrIQd2+auhMVGQEP7l0Av+34DQ27j3M5fd9wue7D/rs+KGupVVZGeD+iDb/dspwoiKkyzkTW/fXsHhrOd84K6tHxeYAoiMj+Gr+SP65tZz9h8Pvg0B3FFfVEh0pjPBBJ39uagK7q+vC5irdmySRq6q/x5lUp6p1gDWA9sDLa/bw5Qc+o6lZeeHWmVw/Y7RPk0N786dk8Orts4iJiuCah1bwQpiUmd5SVsOR+mafLjLkrSHxMcwem8Jb6/fR2sn61w99VER8TCQ3zMzq1fmunZZJS6vyknVgd6q4opbMIQNPGvjREzmp8a5Cf9XhcTXhzSvW6JQFVwARyQUa/BpVP9PQ3MJPX9vID15az+mjkvnHd2Zz+qhO12vymYkZSbx552zOzBnCj1/ZyE9f29jvPwEVOOtO+6uoX1fmT8lg76FjrPVw9VZaXceb6/dx3Zmjet28OHpoPLPGDOX5VaWdJqVwV1JVS3YPq7+2l+MMow2XQn/eJIlfAu8BmSLyDLAY+JE/g+pP9h06xtUPreCZgt3cdm4uT98yPeDjq5PjY3jiG9O57dxcninYzYJHVlDeg5nBfUVBURWjhgxk+KDgzB+4cNIwYqMieMNDZdhHPy4iQuCW2b4Z/7Fg+ij2HjrGxzsrfXK8/qa1VV1Jwgf9EeBeDTY8Rjh5M7rpA+Aq4CbgOSBfVZf6N6z+4dOdlVx23ycUlh/lwa+fzj2X5PnkcrcnIiOEey7J4/7rTmPzvhouu+8Tvth3OCix+FNrEPsj2iTERjFvQjrvbCw7qb5S5dEGnl9VylWnjWTYIN90ql84cRhD42N4riA8mhO7a39NPfVNrb0e/tomMS6atMTYsBnh5M3opreAC4GlqvoPVe3VxxURGS8i69xuNSLyvXb7zBGRw277/Lw35ww0VeWvS3dy/WMFDI2P4Y1vz+LiycODHRYAl52awWt3nkVUhHDDYyv73aSg7eVHOFTXFND5ER25fEoGVbWNfFpYdcL2Jz8robGllYUeyoH3RExUBF8+YyQfbjkQliPZutI2ubGnJcI74qrh1L/+dzzx5mPtH4Czgc0i8rKIfEVEevwRSFW3qepUVZ0KnAHUAa91sOvHbfup6q96er5Aq6lvYuHTa/j9e9u49JThvH7nrOOzNENF3rAk/v5NV7X36x9bSdnhY0GOyHcKipz+iCBeSQDMGZ9KYlwUb6zbe3zb0YZmnvyshIsnDfP538S10zJpblVetmKPJynqZYnwjuSmJlAYJoX+vGlu+khV78A1w/oh4Gpc6137wlygUFV3+eh4QbV1fw3z7/uEJVvL+fllE7lvwWnEx3ozqT3wclITeOIb0zl8rIkbHlvJwX5SsKyguIoRgwf0qtKnL8RFR3LxpGF88MUB6ptcayI/V7CbmvpmbjvXu0J+3ZGTmsCMnCE8v9I6sNsrqawlNiqCYT6cM5PjFPqr7if/N53xqoHcGd30ZeA2YBrwpI/Ofy2ufo6OzBSR9SLyrohM6iS2hSKyWkRWV1QEr3TyG+v28qW/fEZtYwvPfmsGN8/O9uvwVl84ZeQgHrkhn13VdXzjiVXUNjQHO6ReUVVWFge3P8LdFVNHcLShmSVby2lobuHRT4o4K3coUzIH++V8C6aPYnd1HZ+1a+IKd22d1t2pjdWV44X+wmBSnTd9Ei8CW4DzgftxzZu4q7cnFpEYYD7wUgcPrwVGq+oU4D7gdU/HUdWHVTVfVfNTU1N7G1a3NTa38ss3v+C7z69j8ogk3r5rdkBKU/vKzNyh3LfgNDbsOcRtf19DQ3NLsEPqscKKo1QebQza0Nf2ZuYOJSUhljfW7eONz/dxoKbB63LgPXHRpGEMHhjNc2EyH8ZbRZW1ZPlo+GubMU5zYWF5/++X8OZK4jFcieE2VV0CnCUif/HBuS8B1qrqgfYPqGqNqh517r8DRItIig/O6XM/eXUjT3xWwi2zs3n2WzMCXgbCFy6aNIzffflUPt5Ryd0vrKeljzZXrDjeHxHcTus2betf/3NbOX9ZupPJI5KYPcZ/f8Zx0ZF8+fSRfPDFfiqP2lQmcFUDLq2uIzvVt0kiY/AAYqIi7EoCQFXfB04Vkd+LSAnwa2CrD869AA9NTSIyTJy2GhGZ7sQZctfQ9U0tvLOxjAXTM/nZZRN7VKQtVFydn8lPL53A2xvL+Nkbm/pkh9yKoirSk2IZ3cty0L40f2oGjc2t7Kqq4/Zzx/i9CXLB9EyaWtRWK3TsPXSMphb12US6NpERQvbQ8Bjh5LFXVUTG4XojXwBUAi8Aoqrn9fakIhIPXADc6rbtNgBVfRD4CnC7iDQDx4BrNQTftVaVVHOsqYULJqYHOxSf+NY5OVTXNfLA0kKGDIzhBxeND3ZIXlNVCoqrmdmD6rn+dFrmYEYNGUiEwMWTh/n9fGPSEpmWlczzq0pZeE5OSL0WwVDsh5FNbXJS49m6v+tCjn1dZ0NvtgIfA5ep6k4AEfm+L06qqrXA0HbbHnS7fz+u/o+QtmRrBTFREczMCcmWsB750UXjOVTXyP1LdjJ4YDTfPLtvVIUvrqyl4khDyPRHtBERHr9pGlERErA1HxZMH8XdL65nRVE1M3NDo+ktWNrmSPhqtrW73NQEPth8gMbmVmKi+m4rQlc6+8muAsqAJSLyiIjMxQr7nWDptnJm5gztcRXPUCQi/ObKU7hk8jB+8/aWPtNscbxeU4j0R7gbk5bgl0+ynlx6ynCS4qJsDWxcHx4SYqNISYjx+bH/VeivzufHDiUek4Sqvq6q1wJ5wBLge0CaiDwgIhcGKsBQtauqlqLKWs4bH/gRVf4WGSHce+1UZo0Zyo9e2cCizSeNLQg5BZCZFZEAABsvSURBVEVVpCTEkuvjDsq+KC46kqtOH8l7m/aHxTj+zhRX1ZGVMtAvzW45bSOc+nm/hDcd17Wq+qyqXg6MBD4Hfuz3yELc0m2uORlzvFy8vq+JjYrkoevzmZyRxJ3PrmVFUciNGziurT+iO8u99ncLpo+isaWVV9f2jStBfymuPEp2in8qHhyfK9HPazh1qyFNVQ868xLm+iugvmLJtnKyU+ID2owQaAmxUfztG9MZNWQg33pyNZv2hmZBwNLqY5QdrmdGiPVHBNP4YYmcPmowz63c3SdHqvlCY3Mrew8eI9tPo92S4qJJTYzt9yOc+m9vix8da2xheWEVc/phU1N7Q+JjePqW6SQNiOamv608PloklKxw1rMOdlG/UHP9zNEUVtRy499W9av6XN7aXV1Hq+LzORLuclLirbnJnGxFURUNza2c10+bmtobPmgAT90ynVaFrz9aEHJLZRYUVTMkPoaxaaFVSDHYrpw6gl9fMYlVxdVc+OdlvLp2T1hdVRwf/urjORLuctMS+v2EOksSPbB0WzkDoiP7VPmN3spNTeBJpyDg9Y8VcKgudDpEC4qrmJ5l/RHtiQjXz8zi3e+ezfj0RO5+cT23Pr2GiiPhMRvbn8Nf2+SkxHOorn8X+rMk0U2qypJtFZyVO5S46P4z9NUb7gUBf/zKhmCHA7hm1O45eCzk5keEkqyUeF64dSb/cWkeS7dXcNG9y3hnY1mww/K7ospakgdGM3ig74e/tskNgxFOliS6qaiylt3VdczJC4+mpvZm5g7lzjljeP+LA6wvPRTscChwRl2F4vyIUBIZISw8J5e375rNiMEDuOOZtXznuc9D6orQ10oqa/0+sKQtSfTnzmtLEt10fOjruP7fae3JzbOzSB4YzR8XbQ92KBQUVTNoQDR5wxKDHUqfMDY9kVfvOIu7LxjHOxvLuPDPy/jn1tCfB9MTxZW+W9fakxHJTqG/fjwM1pJENy3dVs6YtISgL2oTTIlx0dw+J5dl2ytY6cx0DpaC4iqmZQ3x6VoB/V10ZATfmTuW1++cRfLAGG5+YjU/fnkDR+qbgh2azxxrbGF/Tb3PC/u1FxkhZA0daM1NxqW2oZmCoup+Ocu6u66fkUVqYix/eH9b0EbMHKipp6SqzuZH9NDkEYN4865Z3D4nl5fWlHLxvR/z2c5eLWEfMkqq/FfYr73c1AS7kjAunxVW0dgSPkNfOzMgJpK7zh/DypJqlu0IzhvLCuuP6LXYqEh+fHEeL99+FrFREVz3aAG/eGMTdY3dW6WwtqGZ4spaCoqqeGv9Pv72aXFQ59QUB2BkU5uc1Hh2V9fR1NLq93MFQ2guwByilm4rJz4mkvws++QKcO20UTz0URF//GAb54xNCfgQ1ILiahJjo5iYkRTQ8/ZHp49K5u3vnM3v39/K3z4t4aPtFfzhq1MYm55IxZF6ymsaKD/SQPmReg603a+pp+KI6/7RDpa+/WRHJY/dNC0IP41/S4S3l5OSQHOrsquqjjH9cK6OJQkvqSpLt1Uwa0xKvy4L3B0xURF8d95YfvTyBj7YfICLJvl/vQR3BUVV5GclB6wEd383ICaSX1w+iQsmpvPDlzbwlQeXd7xfdCRpSbGkJ8YxISOJcxNjSUuMIy0xlvSkONKSYvnbp8W8unYvxxpbglIlubiylrTEWBJi/f8Wl5v2rxFOliTC2I7yo+w9dIxvnz8m2KGElKtOG8GDSwv50wfbmTchPWBv2BVHGiisqOWr+ZkBOV84OSs3hfe/fw7PFuxCENKSnCSQFHv8jberq8ZLTxnOcytL+XRnJfOCsChXIIa/tjle6K+fzry2JOGlJVvLAcKiXlN3REVG8L0LxvGd5z7nHxv2ccXUEQE578rj60dY058/JMRGsfCc3B4//8zsoSTERrF464HgJImqWubmBea8SXHRpCTEUljeP0c4WbuJl5ZuqyBvWCLDBw0Idigh57JThpM3LJF7P9xBc4A67wqKqxgYE8nkEYMCcj7TPTFREZwzLoXFW8ppbQ3s6Lea+iYqjzb6tbBfe7mp8f32SsKShBeO1DexqqS6364d0VsREcLdF4yjuLKWVwK0fkFBUTVnjE4mOtL+hEPV3Lx0yo80sGlfYEvMlwSgsF97OakJ/XbWddD+w0SkREQ2isg6EVndweMiIv8nIjtFZIOInB6MOAE+3VlJc6va/IhOXDAxnSmZg/m/xTtpaG7x67mqaxvZduAIM6w0eEg7Ly+NCIEPt5QH9LyBHP7aJjc1noP9tNBfsD+GnaeqU1U1v4PHLgHGOreFwAMBjczN0m0VJMZFcfro5GCFEPJEhB9cOI69h47x/MpSv56rrT/CJtGFtiHxMZwxOpkPA7z8bXFlLSIw2k+LDXWkP9dwCnaS6MwVwFPqsgIYLCLDAx2Eq+prOWePTbGmjS7MHpPC9Owh3L9kJ8ca/Xc1UVBcRVx0BKeMGOy3cxjfmDshnc1lNew7FLhFj0oqa8kYNCCgVZrbRjhtLqsJ2DkDJZjvegp8ICJrRGRhB4+PANw/ku5xtp1ARBaKyGoRWV1RUeHzILeUHeFATYP1R3hBRPjhReOpONLAk8tL/Haetv4Im68S+uZNcP3fLN4auCanQBT2a29k8kDGpiXw27e38G4/K8MezP+y2ap6Oq5mpTtF5JyeHMRZcztfVfNTU33fZ7BkmzP0NYyrvnbHtKwhnDsulQc/KvRLwbjDdU1s2V9jpTj6iNzUBEYPHcjiLYFpclJViitryUoJbAHOyAjh+YUzmJiRxB3PruXRj4v6zSqAQUsSqrrX+VoOvAZMb7fLXsB9ptRIZ1tAfbStgkkZSaQlxQX61H3WDy4cz6G6Jh77pNjnx15VUo2qzY/oK0SEuXnpfFZY1e16UD1RXdtITX1zQEc2tRmaEMtz35rBxZOG8Zu3t/D/3tpMS4CH//pDUJKEiMSLSGLbfeBCYFO73d4EbnBGOc0ADqtqQK/jDtc1sWb3QSvo102njBzERZPSefTjYg76eLTHiqIqYqIimJJp/RF9xbyJaTQ2t/JxAApBtlV/zQngHAl3cdGR/OW60/nm7Gye+KyE2/6+xq/9c4EQrCuJdOATEVkPrATeVtX3ROQ2EbnN2ecdoAjYCTwC3BHoID/eWUFLq3JenjU1dde/Xzie2sZmHlpW5NPjFhRXc1rm4LBbOrYvm5Y1hMS4qICMciqurAMCO0eivYgI4T8vm8gvL5/Ih1sOcO0jK6g82nfXFQ9KklDVIlWd4twmqepvne0PquqDzn1V1TtVNVdVT1HVk+ZS+NuSrRUMHhjN1Ewb+tpd49ITuWJKBk98Vkz5kXqfHLOmvokv9h3mTJsf0adER0YwZ3waS7b5f/b1su2u4eqhsCjYTbOyeejrZ7Btfw1f+uunfXZhIhse4kFrq/LR9grOHptqVUZ76HvzxtHUovx1SaFPjvfexv20Ksyw/og+Z96ENCqPNrJuj//WRa840sC7m8r4yhkjQ2a4+oWThvH8wpkca2zhqr9+5peVHFtblT0H63x+3Dah8UqGoC/21VB5tMFmWfdCVko8Xz1jJM8W7GZvL8bJry6p5rpHVvCjVzaQnRJvkxr7oDnj0oiMEL+Ocnph1W6aWpSvzxjtt3P0xNTMwbx6+yyGJsTw9UcLeGv9Pp8cd3dVHX9atJ2zf7+Eqx9c7rerNEsSHizZVo4InGNDX3vlrrljAbhv8Y5uP3dd6SFueHwlX3lwOdsPHOXnl03k3e+ebf0RfdCggdHkj05msZ9KdDS3tPJMwW5mj0k5Pvs5lIwaOpBXbz+LqZmDueu5z3lgaWGPhsjWNjTz0upSrnloOef8zxLu++cOclLj+fElebT6acitlQr3YOm2ck4dOZiUhNhgh9KnjRg8gOvOHMXTK3Zx67m5Xk1y2rT3MH9etJ3FW8tJHhjNTy7J4/qZoxkYY3+ufdkFE9P5zdtbKK2u83mfweKt5ZQdrueX8yf59Li+NHhgDE/dMp0fvryB/35vK3sO1vH/5k8iqoumsdZWZWVJNS+v2cM7G8uoa2whOyWeH140ni+dNoKMwf6tTG3/dR2orm3k89JDfOf8scEOpV+447xcnl+1m3s/3M7/Xnuax/227q/hz4u28/4XBxg0IJofXjSeG8/KCsjqYsb/5k5wJYnFWw5w06xsnx776eW7yBgUx9y80B6uHhcdyf9eM5WRyQN4YGkhZYfruW/BacR38DdeWl3Hq2v38sraPeyuriMhNor5UzL4yhkjOWN0csCWC7b/vg58vKMCVVcVS9N7aYlx3HRWNg8tK+SOOWMYPyzxhMd3lh/l3g+38/bGMhJiovjevLHcPDubpLjoIEVs/CE7JZ6c1HgWby33aZIorDjKJzsr+cGF47r8VB4KIiKEH1+cx8jkAfzs9U1c8/ByHr9xGmlJcdQ1NvPepv28vGYPnxVWIQJn5Q7l+xeM5eJJw4OyFKwliQ4s2VrO0PgYTrUFbXzmtnNzeGbFLv60aBsPXe8q+ltSWcv/Lt7BG+v2EhcdyR1zcvnW2TkMHhgT5GiNv8ybkM7fPi3mSH0TiT76EPD3FbuIjhSumTbKJ8cLlK+dOZqMQQO489m1fOmvnzFrzFDe2bifow3NjBoykLsvGMdVp49gZHJwh/NakminpVVZtqOSOeNSibChrz4zeGAMt5ydzb0f7uDdjWUs2VbOK2v3Eh0pfOvsHBaek8NQ6//p9+bmpfHwsiI+3lHJpaf0vqhzXWMzL6/ZwyWTh5Oa2Pf+fs7LS+PFW2dy8xOr+MeGMv7tlOF8NT+TaVmBa07qiiWJdjbsOUR1bSPn2tBXn7tldjZPflbC7c+sJSYqghtnZnHbnBzSEq0uVrg4Y3QygwZE8+GWAz5JEm+s28eR+mZumBlaw167Y/KIQXx2z/k0t2pIjtyzJNHOkm0VRAicM9aShK8lxkXzX1edytrdB7l5VjbDBllyCDdRkRGcn5fGkq3ltLRqryaqqipPLd/FhOFJnNHH585ERUYQFXr5AbB5EidZuq2c00Ylkxxv7eL+cPHkYfzHpRMsQYSxuRPSOFjXxNrdB3t1nDW7DrKlrIbrZ4wOmaaZ/siShJuKIw1s2HPY1o4wxo/OGZdKVITwYS9nXz+9YheJsVFceVqGjyIzHbEk4WbZdtfKdjb01Rj/SYqL5sycIb2afV1xpIF3Npbx5TNG2iRLP7Mk4WbJtnJSE2OZODwp2KEY06/NzUtnZ/lRdjnrP3TXi6tLaWpRru/DHdZ9hSUJR3OLa1EUG/pqjP/Nm5AOwIc9uJpobmnlmRW7QrZOU39jScKxrvQQh481McdWoTPG70YNHci49IQeLUS0eGs5+w7Xh1y11/7KkoRjybZyIiOE2WNTgh2KMWFh7oR0VpVUc/hYU7ee9/cVuxg+KI55E+wDXSBYknAs2VpxfKKPMcb/5k1Io9lZ3MtbRRVH+XhHJddNH9Un6jT1B/YqAwdq6tlcVsN51tRkTMBMzUxmSHxMtxYi+vuK3URHCtdO71t1mvqygCcJEckUkSUisllEvhCR73awzxwROSwi65zbz/0Z00fbXJ9k5lgpDmMCJjJCOG98Gku3VdDc0trl/nWNzby0prTP1mnqq4JxJdEM/LuqTgRmAHeKyMQO9vtYVac6t1/5M6Al28oZlhRHXrsS1sYY/5o3IY3Dx5pYvavr2ddvOnWabNhrYAU8Sahqmaqude4fAbYAIwIdR5umllY+2VHJeXmpNrXfmAA7e1wqMZERXY5yaqvTlDcskfw+Xqeprwlqn4SIZAGnAQUdPDxTRNaLyLsi4tc1Cf949RS+dqZ9OjEm0BJio5iRO5TFWzufL7F290E2l9Vw/Uyr0xRoQUsSIpIAvAJ8T1Vr2j28FhitqlOA+4DXOznOQhFZLSKrKyq8HyXRJjoyggsnDWOyLTBkTFDMm5BGcWUthRVHPe7z9HKnTtPUoDU6hK2gJAkRicaVIJ5R1VfbP66qNap61Ln/DhAtIh1OYFDVh1U1X1XzU1Ot49mYvuZ8p1aap1FOlUcbeGfjfr58xsgO14I2/hWM0U0CPAZsUdU/edhnmLMfIjIdV5xVgYvSGBMoI5MHkjcs0WOJjhdWldLY0mozrIMkGGl5FnA9sFFE1jnb/gMYBaCqDwJfAW4XkWbgGHCtqmoQYjXGBMC8Cek88FEhB2sbT1jLpaVVebZgN7PGDGVMmtVpCoaAJwlV/QTotOdJVe8H7g9MRMaYYJs3MZ37l+xk6fZyvnTayOPbF285wN5Dx/jZZR2NkjeBYDOujTFBd+qIQaQmxp7U5PS01WkKOksSxpigi4gQzh+fxrJtFTQ2u2ZfW52m0GCvvDEmJMydkMaRhmZWlVQD8EyBq07TNdMzgxxZeLMkYYwJCbPHphATFcGHWw5wrLGFl1aXcvHk4aQlxgU7tLBmg46NMSFhYEwUs8ek8OGWA4xPT6SmvpkbrE5T0NmVhDEmZMydkEZp9TH+tGi71WkKEZYkjDEhY26ea+3r8iMNVqcpRFiSMMaEjGGD4pg8IsnqNIUQ65MwxoSU//rSqdTUN1mdphBhvwVjTEg5ZaRVZA4l1txkjDHGI0sSxhhjPLIkYYwxxiNLEsYYYzyyJGGMMcYj6U9r+YhIBbCrh09PASp9GI6vWFzdY3F1j8XVPf0xrtGq6nHt536VJHpDRFaran6w42jP4uoei6t7LK7uCce4rLnJGGOMR5YkjDHGeGRJ4l8eDnYAHlhc3WNxdY/F1T1hF5f1SRhjjPHIriSMMcZ4ZEnCGGOMR2GfJETkYhHZJiI7ReSeAJwvU0SWiMhmEflCRL7rbP+liOwVkXXO7VK35/zEiW+biFzkr9hFpERENjrnX+1sGyIii0Rkh/M12dkuIvJ/zrk3iMjpbse50dl/h4jc2MuYxru9JutEpEZEvheM10tEHheRchHZ5LbNZ6+PiJzhvP47ned6teKOh7j+R0S2Oud+TUQGO9uzROSY2+v2YFfn9/Qz9jAun/3eRCRbRAqc7S+ISEwv4nrBLaYSEVkXhNfL03tDcP/GVDVsb0AkUAjkADHAemCin885HDjduZ8IbAcmAr8EftDB/hOduGKBbCfeSH/EDpQAKe22/R64x7l/D/Dfzv1LgXcBAWYABc72IUCR8zXZuZ/sw9/XfmB0MF4v4BzgdGCTP14fYKWzrzjPvaQXcV0IRDn3/9striz3/dodp8Pze/oZexiXz35vwIvAtc79B4HbexpXu8f/CPw8CK+Xp/eGoP6NhfuVxHRgp6oWqWoj8DxwhT9PqKplqrrWuX8E2AJ0tgTXFcDzqtqgqsXATifuQMV+BfCkc/9J4Eq37U+pywpgsIgMBy4CFqlqtaoeBBYBF/solrlAoap2Nqveb6+Xqi4Dqjs4X69fH+exJFVdoa7/5qfcjtXtuFT1A1Vtdr5dAYzs7BhdnN/Tz9jtuDrRrd+b8wn4fOBlX8blHPdq4LnOjuGn18vTe0NQ/8bCPUmMAErdvt9D52/YPiUiWcBpQIGz6dvOZePjbpeonmL0R+wKfCAia0RkobMtXVXLnPv7gfQgxNXmWk785w326wW+e31GOPd9HR/Azbg+NbbJFpHPReQjETnbLV5P5/f0M/aUL35vQ4FDbonQV6/X2cABVd3hti3gr1e794ag/o2Fe5IIGhFJAF4BvqeqNcADQC4wFSjDdckbaLNV9XTgEuBOETnH/UHn00dQxkw77c3zgZecTaHwep0gmK+PJyLyU6AZeMbZVAaMUtXTgLuBZ0Ukydvj+eBnDLnfWzsLOPGDSMBfrw7eG3p1vN4K9ySxF8h0+36ks82vRCQa1x/BM6r6KoCqHlDVFlVtBR7BdZndWYw+j11V9zpfy4HXnBgOOJepbZfY5YGOy3EJsFZVDzgxBv31cvjq9dnLiU1CvY5PRG4CLgO+5ry54DTnVDn31+Bq7x/Xxfk9/Yzd5sPfWxWu5pWodtt7zDnWVcALbvEG9PXq6L2hk+MF5m/Mmw6V/nrDtcZ3Ea6OsrZOsUl+Pqfgagu8t9324W73v4+rfRZgEid26BXh6szzaexAPJDodv8zXH0J/8OJnWa/d+7/Gyd2mq10tg8BinF1mCU794f44HV7HvhGsF8v2nVk+vL14eROxUt7EdfFwGYgtd1+qUCkcz8H15tEp+f39DP2MC6f/d5wXVW6d1zf0dO43F6zj4L1euH5vSGof2N+ezPsKzdcIwS24/qE8NMAnG82rsvFDcA653Yp8DSw0dn+Zrt/pp868W3DbTSCL2N3/gHWO7cv2o6Hq+13MbAD+NDtj02Avzjn3gjkux3rZlwdjztxe2PvRWzxuD45DnLbFvDXC1czRBnQhKs99xZfvj5APrDJec79OBURehjXTlzt0m1/Yw86+37Z+f2uA9YCl3d1fk8/Yw/j8tnvzfmbXen8rC8BsT2Ny9n+BHBbu30D+Xp5em8I6t+YleUwxhjjUbj3SRhjjOmEJQljjDEeWZIwxhjjkSUJY4wxHlmSMMYY45ElCWMcInLU+ZolItf5+Nj/0e77z3x5fGP8xZKEMSfLArqVJNxm/npyQpJQ1bO6GZMxQWFJwpiT/Q4421k/4PsiEimu9RlWOYXpbgUQkTki8rGIvIlrdjMi8rpTIPGLtiKJIvI7YIBzvGecbW1XLeIce5NT5/8at2MvFZGXxbUuxDNttf9F5HfOmgMbROQPAX91TFjp6tOPMeHoHlxrHlwG4LzZH1bVaSISC3wqIh84+54OTFZXeWuAm1W1WkQGAKtE5BVVvUdEvq2qUzs411W4it1NAVKc5yxzHjsNV7mKfcCnwCwR2QJ8CchTVRVnMSFj/MWuJIzp2oXADeJarawAV5mEsc5jK90SBMB3RGQ9rjUcMt3282Q28Jy6it4dAD4Cprkde4+6iuGtw9UMdhioBx4TkauAul7/dMZ0wpKEMV0T4C5VnercslW17Uqi9vhOInOAecBMVZ0CfA7E9eK8DW73W3CtNNeMq3Lqy7gqvL7Xi+Mb0yVLEsac7Aiu5SPbvA/c7pRxRkTGiUh8B88bBBxU1ToRycNVbbNNU9vz2/kYuMbp90jFtbTmSk+BOWsNDFLVd3BVUZ3SnR/MmO6yPgljTrYBaHGajZ4A/hdXU89ap/O4go6XfXwPuM3pN9iGq8mpzcPABhFZq6pfc9v+GjATV/VdBX6kqvudJNORROANEYnDdYVzd89+RGO8Y1VgjTHGeGTNTcYYYzyyJGGMMcYjSxLGGGM8siRhjDHGI0sSxhhjPLIkYYwxxiNLEsYYYzz6/wESrbj2RxrGgwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x0-XQ5DMGFkX",
        "colab_type": "text"
      },
      "source": [
        "#### Section 3.6 Price American Option"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WhK5tPofGcCO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "f43c54b0-05f4-4eff-90bb-24ca79dbf33b"
      },
      "source": [
        "# save policy\n",
        "import os\n",
        "import tempfile\n",
        "from tf_agents.policies import policy_saver\n",
        "\n",
        "tempdir = os.getenv(\"TEST_TMPDIR\", tempfile.gettempdir())\n",
        "policy_dir = os.path.join(tempdir, 'policy')\n",
        "tf_policy_saver = policy_saver.PolicySaver(agent.policy)\n",
        "tf_policy_saver.save(policy_dir)\n",
        "\n",
        "saved_policy = tf.compat.v2.saved_model.load(policy_dir)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/tracking.py:111: Layer.updates (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "This property should not be used in TensorFlow 2.0, as updates are applied automatically.\n",
            "INFO:tensorflow:Assets written to: /tmp/policy/assets\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NuKzu-YR3MAy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "039b1384-fabd-4c9b-f751-4f9d53aaabba"
      },
      "source": [
        "# Monte Carlo simulation -- takes a while\n",
        "# npv = compute_avg_return(eval_env, saved_policy, num_episodes=2_000)\n",
        "npv = compute_avg_return(eval_env, agent.policy, num_episodes=2_000)\n",
        "pricing_dict['ReinforcementAgent'] = npv\n",
        "print(npv)"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "7.057035\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aXz88SHGBhy5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        },
        "outputId": "a926d632-aeaa-4c59-ffe3-7aa86b22c42a"
      },
      "source": [
        "import pandas as pd\n",
        "pricing_df = pd.DataFrame.from_dict(pricing_dict, orient='index')\n",
        "pricing_df.columns = ['Price']\n",
        "pricing_df"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Price</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>BlackScholesEuropean</th>\n",
              "      <td>6.927869</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>BawApproximation</th>\n",
              "      <td>7.091255</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>BinomialTree</th>\n",
              "      <td>7.090925</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ReinforcementAgent</th>\n",
              "      <td>7.057035</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         Price\n",
              "BlackScholesEuropean  6.927869\n",
              "BawApproximation      7.091255\n",
              "BinomialTree          7.090925\n",
              "ReinforcementAgent    7.057035"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8aWKrv4CV4z0",
        "colab_type": "text"
      },
      "source": [
        "#### TODO\n",
        "\n",
        "Some [variance reduction](https://en.wikipedia.org/wiki/Variance_reduction) techniques can be applied to improve simulation efficiency.\n",
        "\n",
        "1. Use another AmeriOption gym environment to provide [Antithetic variates](https://en.wikipedia.org/wiki/Antithetic_variates)\n",
        "\n",
        "2. In function compute_avg_return, continue the simulation path to price European option as [Control Variates](https://en.wikipedia.org/wiki/Control_variates)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ETrCNGP-VaEP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}